Determine which type of classical music you are interested in. A Baroque piece can sound very different from a Romantic piece. Do some research as to what styles and composers you're interested in before making any big purchases. The best way to do this is to listen. As classical music is public domain music, countless orchestras and musicians have performed these pieces, so it should be easy to find samples on the Internet for your review. Knowing what era of classical music you're interested in can make things easier when you come across a dealer or see music labelled according to era.
Attend record fairs and swap meets that feature classical vinyl music recordings. Visit Goodwill and Salvation Army locations as many contain older records donated by estates that include classical vinyl music recordings. Talk to people involved in local orchestras and ask them where they get their records.
Pay attention to the record labels on the classical vinyl music recordings. They aren't classified the same way pop or rock recordings are. They are classified by record label, not by composer or performer. Look out for specific labels. Some labels are known for their superior sound and releases. Decca's SXL series, EMI's ASD series, Columbia's SAX series, Mercury and RCA are all all quality recordings and highly sought after.
Don't let the price of a classical vinyl music recording deter you from or sway you into a purchase. The price of a record has nothing to do with the quality of the performance or the quality of the sound. A cheap record can have some of the best quality music on it, while a marked-up record might be just the opposite. Buying cheaper records will give you the opportunity to buy more and expand your collection and classical music knowledge. Don't be afraid to get two of the same pieces of music performed by two different orchestras. Each recording brings with it its own unique take on classical pieces and offers a new listening experience.