Obi-Wan lived as a hermit on the desert planet Tattooine during the 20 years between "Revenge of the Sith" and "A New Hope," but he wasn't languishing in despair waiting to die. He was telepathically in contact with the Jedi Master Yoda the whole time he continued his training. Although Obi-Wan was himself a Master, the rank doesn't mean that Jedi have nothing further to learn. In fact, Yoda was teaching Obi-Wan how to retain his identity and consciousness after death via a strong connection and complete surrender to the Force. The technique is informally called a "Force Ghost." Obi-Wan's disappearance at death is a certain indicator of his becoming a Force Ghost, along with his later appearances to Luke and Yoda.
Yoda also knew the Force Ghost technique, as shown in "Return of the Jedi." Yoda himself received training from the late Qui-Gon Jinn, Obi-Wan's Master who died in "The Phantom Menace." Although Qui-Gon's body did not vanish, he had such a strong attunement to the Force that the Force allowed him to maintain his identity after death. Obi-Wan was the first Jedi in the films seen to perfect the technique, allowing his body to disappear.
The exact requirements of becoming a Force Ghost are unknown. Master Jedi such as Yoda and Obi-Wan became Force Ghosts because of their strong connection to the Force, but other Jedi, such as Mace Windu or many of the Jedi that died during the Battle of Geonosis, did not become Force Ghosts during the scope of the films. Qui-Gon followed the Living Force, or the aspect of the Force that embraced all living things, which allowed him to maintain his essence. Obi-Wan later learned how to listen to and embrace the Living Force during his time on Tattooine.
The Sith and Dark Jedi constantly sought immortality, but their philosophies on the Force prevented them. The Sith and Dark Jedi believed in empowering themselves and using the Force to serve their own ends rather than following its will, whereas the Jedi espoused a philosophy of compassion, selflessness and surrender to the Force's will. Because the Force surrounds and controls everything in the "Star Wars" universe, Jedi can become completely one with it through meditation. Although Obi-Wan had his moments of skirting the Dark Side such as the fight with Darth Maul in "The Phantom Menace," he was still calm enough overall to listen to the Force.