Sort through the used books, checking the front pages for the date of publication. Place any paperbacks published in the last few years into a separate pile, unless they are signed by the author.
Examine the reverse of the title page of each of the remaining books. Make a note of the author, publisher and edition. Look for publishing history, usually at the bottom of the page. If this is absent, the book could be a first edition. If the publishing information has a "1" at the beginning or end or the string of numbers, this indicates a first edition.
Note whether the book is a paperback or hardback. If it is a hardback, note whether it still has the dust jacket.
Evaluate the condition of the book. Look for foxing (browning and spots), tears, dog-ears, pencil or pen marks and dents. Booksellers describe the condition of books that look fresh and bright without any signs of wear as "Fine." Very minor wear indicates the book is in "Very Good" condition. A few obvious minor flaws such as folds on one or two pages, categorizes the used book as "Good." Numerous flaws, heavy wear or minor damage constitutes a "Fair" ranking. For example "Fair" means the book might be tattered, but it is still readable. "Poor" indicates the book is in bad condition, but still possible to read.
Evaluate the condition of the dust jacket using a similar rubric listed in Step 4.
Search for your edition on a website such as Abebooks or Bookfinder. Amazon and eBay also show results for used books. Compare all the prices for that book. Realize that some books are over-valued so don't go by the highest prices. Click on a few descriptions of the book's condition. For example, if the sellers are asking an average of $50 for books in "Very Good" condition and yours is merely "Good," lower the price accordingly.
Consult a local second-hand book dealer, if a book didn't appear in your search or the prices were wildly ambiguous. Dealers have the expertise to give you an indication of a book's value. Small business owners are often happy to provide advice or make an offer for the book.