Know your market. Even if you've got the most compelling story idea in centuries and an expertly written query letter, sending a science fiction story to a magazine about WWII history won't get you anywhere. Be sure you've researched your market, and read copies of your preferred magazine to make sure your style meshes with theirs. The same is true of book publishers. Be sure they publish the type of book you're writing.
Craft your opening. A query letter should sell your story idea. Whether it's fiction or non-fiction, you want to craft a concise, meaningful description of your story.
Include details about your story. Don't extol the virtues of the piece you've written, just give details. Reassure the editor that your piece fits within their word count guidelines. Let them know if you have pictures or illustrations to go with your story. Also, let them know about any research you did.
Talk about yourself. Don't go overboard. Unless your job or hobbies are relevant to the story, don't include information about them. Simply list one or two relevant credits.
Send it off. Be sure you follow submission guidelines when you mail your query letter. Some editors won't accept emails, while others will. Some might request a self-addressed, stamped envelope for their reply. Following guidelines is critical to your success.
Follow up. Things happen. Your query might have slipped through the cracks, so it's OK to follow up. Be sure to wait for the amount of review time specified in the submission guidelines to pass before you follow up. Then send a short, polite email. Specify what the query was about and when you sent it. The most important thing to remember when following up is to be polite. Even a good idea can be rejected because of bad manners.