The episode is "The Ultimate Computer" (Star Trek: The Original Series, Season 2, Episode 24).
Spock instructs the rogue M-5 multitronic computer to compute pi to the last digit—an impossible, infinite task—causing the system to overload and shut down, saving the Enterprise.