The rarest books these days probably are the few surviving scrolls from the great library at Alexandria. The value of these is beyond measure because each is unique. The same can be said of the hand-copied volumes the monks made in the Middle Ages. Many of them have inlaid golden and elaborate artwork. The first printed books, among them the famed Gutenberg Bible, also are rare and collectible, if very expensive.
While today any book can be copied, printed and rebound, there is nothing like the look, feel and aroma of rare old books. They are valued pieces of any book collector's library. They represent more than what is within their pages; they represent a moment in time and everything about the place, the people and the culture in which they were printed. They are windows into the past.
"Old" does not make a book rare or even collectible. One author, Harvey Chalmers II, had a habit of signing nearly every copy of his historical novels. One bookseller, the late Ken Dorn, used to joke that if you could find a copy of one of these books unsigned, it would be worth more. The content of old books, along with their physical condition, is more important than the age or scarcity.
Gutenberg perfected the printing press in the 1500s. Since then, books have gotten cheaper and easier to produce. Even by the 1800s, however, books still were printed in short runs. It wasn't until the 20th century that books found a mass market, following on the heels of the pulp magazine of the 1890s to the mid-1950s. That is when paperbacks came into widespread use. Even then, however, some books became rare. For example, when Donald Wandrei and August Derleth founded Arkham House to get the works of horror writer H.P. Lovecraft into print, nearly each of the very limited run books was rare nearly from the beginning. Today most of them fetch high prices.
It takes some work to identify rare books. A copy of Lovecraft's "Dagon and Other Macabre Tales" is a bit rare, depending one when the edition was printed, but a copy of "The Outsider" in hard cover from Arkham House is much sought after. The latter was the first appearance of Lovecraft in hardcover, and only a couple thousand copies were printed. Aside from study and immersion in the field of rare books, seeking knowledge and help from dealers and collectors can help you learn to be discriminating. Most antiquarian book dealers have a great deal of knowledge about rare books and are willing to share it.
Beware of shady dealers. There aren1t many, but once in a while, antiquarian book dealers will try to pawn off something as a rare book when it isn't the case. For example with the aforementioned "Outsider," some dealers have been known to try to get higher prices for it by selling "signed" copies. The book was printed and released in 1939, two years after Lovecraft died, so "signed" copies would've been impossible to create.