Braces are used to group a set of items together, such as a set of items to choose from, just as they are used in mathematical equations, to group a set of numbers together and indicate that they are related to one another.
In contrast, parentheses are used to explain or modify a thought or add a personal comment to a preceding statement, or to clarify an idea that has been presented, and the correct usage of square brackets would be to enclose interpolated matter; that is, words that are to be added to the original text.
When faced with the choice of which type of punctuation to use, remember that parentheses are used to express related thoughts that the text could stand alone without, if the information within the parentheses were removed, and square brackets are used to insert information that helps to clarify or be more specific about an idea. Braces however, are used to enclose information that is essential to understanding the choices available.
For example: "This article may help a writer to decide which form of punctuation {braces, parentheses, brackets} is the correct one for the situation." While the example talks about choosing a form of punctuation, the information enclosed within the braces lets you know exactly what the choice consists of; information that could be vital in making a correct decision. Once the purpose for each type of information is determined, it becomes obvious what type of punctuation should be used, according to the type of information that the punctuation marks are to contain. Braces should be used only to illustrate a group or set of items that are related and that are to be included within a single category.