Why do they want to kill Prospero in The Tempest?

In Shakespeare's "The Tempest," Prospero is not being hunted nor is anyone trying to kill him. In fact, it is Prospero who seeks revenge against his brother Antonio and the others who usurped him and banished him from Milan.