Check the functionality of all microphone and instrument cables, and repair or replace those that aren't in full working order. Using cables that work intermittently, or add clicks, pops and crackles to your audio signal chain is a sure way to make your home-studio recordings sound amateurish.
Replace old and worn guitar strings on all guitars you'll be using in your home studio. Getting guitars to sound their best before recording them is preferable to spending valuable time working on brightening their sound with equalizers and other effects, at the mixing stage. Similarly, replace old and worn drum heads on the drum kit, and fix any loose and rattling metal hardware that may be picked and amplified by microphones.
Tune up all stringed instruments and drums. A flawlessly played and recorded performance can be compromised if instruments aren't in tune with each other. It's easy to get caught up in the energy of a recording session and neglect regular tuning checks, but it's a necessary part of every session.
Arrange and rehearse the material before recording it. Spending time polishing your songs until you settle on arrangements that make the most of their potential is crucial to the quality of the recording. A song with a lifeless arrangement will sound dull, no matter how great the equipment or sound engineering is.
Experiment with microphone choice and placement. Choosing the correct microphone and setting it at the optimal distance to a sound source will improve the sound of your recordings more than any rack of effects processors will. Even small adjustments in microphone placement can yield wildly different results, so take your time with it, and make detailed notes for future reference.
Record your tracks without effects. Adding effects like reverberation, compression or delay may sound good initially, but if you record them to the track, you're stuck with them. If, after numerous listens, you find that the effects were overdone, or the wrong effect was selected for that particular song, there is no way to remove that effect. A better route is to record without effects -- which recordists call dry -- and then add effects to taste, when mixing your song. You'll then have more control over the quality of your recordings.
Use the panorama controls of your recording software to move your recorded tracks around in the mix. Turning the pan control knob fully counter-clockwise will place the instrument totally in the left channel, while turning it fully clockwise will place it in the right channel. If an instrument gets lost in a muddy mix, sometimes a slight movement in its panning is all that it takes to create its own space in the stereo field. This is preferable to simply reaching for the volume control, since the added volume level can overload the mix, making it sound harsh and distorted.
Use a graphic equalizer on recorded tracks to shape their tonal character and create space for them in your mixes. The more instruments you have in your songs, the easier it can be for them to get lost in the mix, especially if they're competing with other instruments that are strong in similar frequencies. Use of equalization can give separation to tracks by either removing intrusive frequencies that do nothing to enhance the sound quality of the track, or by adding frequencies that give them a little definition.