In the context of the play, Hamlet is speaking about his own actions and how they may appear to others. He knows that he is acting strangely, but he has his own reasons for doing so. He is trying to avenge his father's death and he believes that the best way to do this is to act insane.
The phrase "though this is madness yet there is method in it" can be used in a variety of situations to describe something that may seem strange or illogical, but actually has a purpose behind it.