No, a standard modem cannot function as a musical instrument in any practical sense. Modems (modulator-demodulators) are designed exclusively for data communication—converting digital signals for transmission over telephone/cable lines and vice versa.
They lack:
However: The characteristic "handshake" sounds of dial-up modems have been sampled in electronic music as percussive/noise elements. Some experimental artists have hacked modem firmware to generate tones, but this is highly specialized, not a standard musical application.