Gather together your entire collection of sheet music. Decide on an organizational strategy, depending on what's contained in your collection. If you exclusively deal with jazz music, for example, you might consider sorting by subgenre, and within each subgenre, sorting artists alphabetically.
Separate the sheet music into piles based on type. If you deal with a variety of music, each type could represent a genre. If you're a music instructor, you might sort the sheet music based on difficulty.
Alphabetize the individual sheet music piles based on either the artist's last name or the title of the piece, whichever is more convenient and will make locating a piece simpler for you. Place each of these piles into an individual folder. If your collection is large, you may even need a separate folder for each letter of the alphabet.
Purchase a filing cabinet or crates in which you'll store your folders. Keeping the folders in a pile on your desk won't be much of an improvement over scattering the pieces of music around your office. The goal is to create a system that you can navigate easily. Sheet music can be expensive, so try to find a locking filing cabinet.
Consider backing up your sheet music collection electronically. Depending on the size of your collection, this may require a sizable initial time commitment, but will be worthwhile considering the level of protection it offers. Whenever you acquire a new piece of music, use a scanner to copy it to your hard drive. Organize your files into folders just as you would with a filing cabinet. Alphabetization by title will be automatic with this system.