The line appears in "Everybody's Doin' It" by Bobby Bare (1972). The relevant verse:
Everybody needs somebody sometimes
Even though it's only for one night
Everybody wants somebody to hold onto
When the morning brings the light.
The song's wry, observational tone typifies Bare's storytelling style. While the exact phrase "look down on" isn't in the chorus, the sentiment of seeking connection—even temporarily—aligns with the theme.