Start with fiberglass batt insulation. This is likely the cheapest option, especially when mounting speakers in an existing wall that may already have insulation. Batt insulation will reduce some of the vibrations through the wall, but it will not perform as well for low-end frequencies. If you are mounting speakers with large woofers that will reproduce very low sounds, you should opt for more than just batt insulation.
Try creating your own acoustic foam using different materials. Standard egg-crate foam works well as an acoustic absorber. Use a serrated knife to cut the foam to size. You can cut multiple pieces and put them together so that the peaks of one piece fit into the valleys of another piece to increase thickness. Wrap the foam with fabric, such as cotton or wool, to make it slightly thicker if necessary. Consider other materials such as foam board insulation, heavy grade Styrofoam or even packing peanuts wrapped in fabric.
If you need more specialized foam, buy premade acoustic foam. Many companies, such as Auralex, make specialty acoustic foam. This foam will cost more, but it is designed to reduce certain frequencies. You can buy custom foam for whatever frequencies you have the most problems with. Cut the foam to size and fit it around your in-wall speaker enclosure.
Separate the in-wall speaker from structural points of the wall. The foam is meant to be an acoustic isolator, but if the speaker enclosure is touching a wall stud, the vibrations will be transferred regardless of the foam. Be sure that your in-wall speaker is separated from the walls as much as possible by the use of acoustic foam.
Verify the sound quality before you finalize your speaker mounts. Run some audio through your in-wall speakers to check if your acoustic foam is having the desired effect---before you replace all of the screws. Listen for frequency balance and any noticeable rattle. If the sound quality is not to your liking, fix it now before you assemble or reassemble the in-wall speaker.
Adjust your foam as necessary after you check your in-wall speaker with an audio source. You may find you need a higher density foam, or you may need to pack more of it in the wall. Because the foam will be hidden behind the speaker, it does not have to be aesthetically pleasing visually. Cut it up and pack it in. Do whatever it takes to get the right sound. Remember that your ears are the final arbiters.