Pause in your speech rather than saying things like "um," "er," and "uh." A pause is much easier for an audience to hear than filler syllables. You can say the "um" in your head if that helps you, but don't let it out of your mouth.
Don't begin your speech by apologizing for your lack of adequate preparation. Even if you really weren't able to get a great speech together, you don't need to notify your audience. Doing so will start things negatively and make errors more obvious.
Don't fidget, jiggle, or play with objects like keys, coins, or your notes. Doing so directs valuable energy from your speaking and is a distraction to your audience.
Connect with your audience and have a conversation with it. While you must look at your notes from time to time, be sure that you know every aspect of your speech well so that you can spend a good bit of your time looking at the people in your audience. It's disappointing to expect a speech but instead get monotonous mumbling and a good look at the top of somebody's head.
Don't use jargon or "high language" that's unknown to your audience. Some speakers, especially beginners, think it's impressive, but it's not. In fact, you can alienate your audience. If you must use a "big word," explain it without appearing condescending.