Check for all basic information about the antique book: author, title, publishing company, date of publication. Check for both the original copyright date and the latest date of printing. It is rare for a novel that is not a first edition to be worth anything, but nonfiction books, especially those on obscure subjects, may have value based on their content.
Research further if any of the above information is missing. If the book appears to be very old and there is no date of publication listed, check first that you haven't missed a date in Roman numerals. If there is truly no date, check online databases like the Library of Congress or used-book search sites to find a match for your book. If the information is missing because the book is incomplete--for instance, because the title page has been ripped out--the book is probably not valuable.
Determine the condition of your book. Examine the book carefully for any flaws, including normal yellowing and brittle pages that you would expect in an old book. Even if the book looks very well preserved for its age, these are still considered flaws; you cannot call a book "fine" or "near mint" unless it looks like it could be sold as new on the shelves of a bookstore. In general, "very good" means a book that is nearly flawless, while "good" covers most normal books without major flaws such as rips, broken spines or water damage.
Identify the edition of the book. To be valuable as a first edition, a book must be from the first printing as well. Look for a series of numbers on the title page that looks like this: "10 9 8 7 6 5 4 3 2 1." In general, the lowest number that is present represents the printing number.
Find out whether the book is a reprint. It used to be common for books to be reprinted from the original plates; this means that a book may carry a copyright date from 1880, for example, even though it was reprinted in 1930. Look for a publisher's imprint at the bottom of the title page that is different from the original publisher, or for any indication that the book was published by Grosset & Dunlap, a popular reprint publishing house.
Determine the book's value. While finding out a rare book's true value is a job for an expert, any amateur with Internet access can usually make a good approximation. Take all the information you have gathered--the publication date, the edition, the condition of the book--and search on Internet bookselling sites like Abebooks, Alibris or Addall for booksellers offering a similar book. If you can't find anything like it, especially if you can't find anyone else selling a first edition or a book in as good a condition as yours, take it to a professional book dealer for an estimate of its value.