Consider the subject matter of the drawing. The main goal when matting any piece of art is to create contrast between the art and the mat. If your pencil drawing is dominated by darker tones, your mat should be lighter. Conversely, light pencils could benefit from darker matting, possibly even black. Drawings using very soft pencils almost look black, and may benefit from a completely white mat.
Consider the size of the drawing. The standard procedure is that the smaller the drawing, the larger the mat in relation. Because of this, really small drawings tend to have quite a bit of matting around them. Make sure the color you pick for the mat is in high contrast to the drawing so the drawing doesn't get lost in all the matting.
Think about where you'll be hanging the drawing. One of the main reasons to matte a piece of art is to separate it aesthetically from the wall it's hanging on, so the worst thing would be to choose matting the same color as the wall. A wall that is completely white can almost always benefit from completely black mat.