The events you might be thinking of are:
* Harry, Ron, and Hermione leave Privet Drive: They leave to start their quest to find the Horcruxes in the beginning of the book.
* The Dursleys are whisked away: After Harry, Ron, and Hermione leave Privet Drive, the Dursleys are taken to an undisclosed location by the Ministry of Magic for their own protection. This happens after the Order of the Phoenix raids Privet Drive and the Death Eaters attack.
It's understandable to get confused, as the Dursleys are absent for a significant portion of the book. However, they never actually leave Privet Drive on their own accord.