Good ad-libs fit into a song seamlessly, so they are almost part of the music. To ad-lib, you need to learn how to find the rhythm and melody of the song. Fit your ad-libs in as part of the melody, or as a counter to the melody. Learn the chords so you can harmonize with the lead vocals, or add punch to the background during rests. The style and type of music will help you figure out which type of ad-libs will work best.
Ad-libbing is meant to complement and enhance the lead vocals, not cover them up. The ad-libs should mesh with the lead. Fill in where more flavor is needed, but don't keep ad-libbing just to be singing. Good ad-libs will make a song more exciting. Too many ad-libs will make a song sound busy. Not enough ad-libs can make a song boring, with not enough punch. If you are recording, it is better to start off with a few ad-libs and add more as needed.
Effective ad-libs are usually short. Many times the ad-libs are not even words, but are just syllables. Scat singing is all randomly placed syllables meant to replicate the sound of a musical note. Something as simple as "Oh, yeah" is frequently used for an ad-lib, as is "Baby." Some artists, such as Michael Jackson, made short guttural ad-libs almost a trademark, and sometimes ad-libs are variations on a song's lyrics, such as the ad-libs around the name "Jude" at the end of the Beatles' song, "Hey Jude." Ad-libs are often repeated several times. Short words emphasizing the lyrics help bring out the meaning of the song.
Once you are familiar with the process of ad-libbing and improvisation, you can do it by ear. Until you get to that point, you need to practice. Practice learning chords, learning harmonies, and learning rhythm. Start out by clapping along to songs in the spots where you would put the ad-lib to learn how to match the song's tempo and beat. Practice singing different ad-libs and working out what type of words will work with what song, so you have a few in your head to call on when you need to improvise.