The scene in Star Wars: Episode IV – A New Hope where Luke Skywalker, Han Solo, and Chewbacca find themselves in the garbage compactor is iconic, but it doesn't feature any specific monster. The danger comes from the compactor itself, the various pieces of junk, and the potential for getting crushed.
It's possible that you are thinking of a fan theory or a story from a non-canon Star Wars source. There are countless fan-made stories and creations in the Star Wars universe, but none are officially recognized as canon.