Determine the demand for the book. Many old and antiquarian books have no market, making them neither rare nor valuable. If a book is a well-known title, or by a popular author, it is more likely that there will still be interest in the book today. Often, there is little to no interest in old religious books, textbooks and science books.
Check the edition of the book. In general, first edition and limited edition books will fetch a higher price than more recent editions, especially if the book or author is well known. In order to determine whether a book really is a first edition, you may have to take it to a book dealer. Terms such as "first printing," "First edition, second printing" or "first issue" can often be misconstrued to mean first edition, which is not the case.
Look for the author's signature inside the cover. Signed copies of books are highly prized by book collectors due to their limited supply. Finding a signature by the author will also increase the value of a book substantially. Check with online book sellers such as Bookfinder, Amazon.com and Abe Books to determine how the signature will specifically effect the price of the book.
Inspect the physical characteristics of the book. Factors that increase the value of a book include special illustrations and typesetting, a rare cover and an old edition still in good condition. Anything that makes the book special or desirable compared to other copies will help it to fetch a better price and determine if it is indeed rare.
Consult with a book seller to determine the value of the book. People who own and operate secondhand bookstores and exchanges will often be more clued in with the demand for a book and its worth. You do not have to sell them any books; simply contact them and ask them for their professional opinion. They will likely want to inspect the book, or refer you to someone more knowledgeable. Other good sources of information are web sites such as BookFinder and AbeBooks, where you can search specifically for rare books and compare prices.