Charlie's father and mother initially choose to send Charlie with his grandparents (Grandpa Joe, Grandma Josephine, Grandpa George, and Grandma Georgina), but in the end, only Charlie and Grandpa Joe go. This is because the other grandparents are too old and frail to make the journey.
This was a decision made by Roald Dahl himself. Grandpa Joe, though bedridden, possesses a certain childlike wonder and excitement that makes him a perfect companion for Charlie. He is also, according to Dahl, the only grandfather who actually believes in the existence of Willy Wonka's chocolate factory.
The film adaptations of the story often make changes to the story, sometimes including Grandpa Joe on the trip to the factory. This is a creative choice of the filmmakers and not necessarily reflective of the original book.
So, in the original book, it's because the other grandparents are too old and frail to go, and Grandpa Joe, despite being bedridden, is the only one who believes in the factory and can share Charlie's excitement.