Look for an agent in the right places. A legitimate literary agent probably won't be advertising in a local newspaper or on a work-at-home website, and she won't solicit manuscripts through Internet message boards or private messages to forum members. If you're not familiar with a literary agency, check to see if they are a member of the Association of Author's Representatives or look for their name in a reputable source like "Writer's Digest."
Check the agent's story. How long has the agency been in business? Who have they represented, and what publishing houses have they sold to? If you can't find the agency's name attached to any projects, it's probably not a good bet. Start-up solo agents should have a track record working at another literary agency.
Ask questions of your literary agent. What was their biggest success, and why? Can he tell you about a project he represented that failed? (This is important -- even the best agents have one or two turkeys in their record, and they should be honest about why.) Ask if the agency will put you in touch with one of its published authors, so you can ask a fellow writer firsthand about his experience.
Know how the industry works. Any agency that wants you to pay them to represent you is not a legitimate agency. Your agent can ask you to edit or change your manuscript, but she shouldn't demand that you pay for another editor to "fix" your work. And don't be swayed by flattery -- a genuine agent will always have constructive criticism and realistic expectations for you, so don't trust someone who swears you're a genius and promises your book will make millions of dollars.