Finding rhymes for penitentiary means working with the -iary ending (pronounced /iˈɛri/ or /iˈɛrɪ/). This is a large family dominated by adjectives and nouns ending in -ary or -ory, but stress patterns vary—so you'll need to check meter carefully when writing verse.
While these words share the terminal sound, their stressed syllables fall in different places. Penitentiary stresses the third syllable (pen-i-TEN-shary), whereas secretary stresses the second (SEC-re-tary). In strict poetic forms, that mismatch can create a rhythmic hiccup. For free verse or song lyrics, the sonic echo is usually close enough to work beautifully.
Group words by stress pattern when building a rhyme bank. For example, arbitrary, complimentary, necessary, stationary all share a similar dactylic feel, making them more interchangeable in lines of poetry.