Decide what type of performer you are. Different styles of magic bring with them different types of props.
Regardless of the style, you need to learn and understand whatever style you choose. Street magicians usually only use cards and coins. Table magicians tend to perform more elaborate tricks.
Make a list of what you need. Decks of cards are essential, no matter what, so be sure to add them to your list.
Magic shops, though rare nowadays, are great for beginners and stage magicians.
The problem with magic shops is that most are privately owned. This means the owners may tend to overcharge or push more items than you need.
Search online! That's what the web is for! Websites like Ellusionist and Penguin Magic allow you to shop at your own will and for cheaper prices. (see links below)
Turn everything into a positive. Do not get angry if you end up with 10 extra props from a magic shop-learn how to use them!