What is the name of glass balls in Harry Potter 5?

The glass balls you're thinking of in Harry Potter and the Order of the Phoenix are called "Memory Balls" or "Pensieve Balls."

They are used with the Pensieve, a magical basin-like object that allows the user to view memories by pouring them into the liquid inside the basin. The memories appear in these glass ball shapes within the Pensieve.