What happens when you use a max repel in the trophy garden?

You can't use a Max Repel in the Trophy Garden. Max Repels are items used in the main game to prevent wild Pokémon from appearing, but the Trophy Garden is a special area where you can only encounter Pokémon that you have already caught and registered in your Pokédex.

There are no wild encounters in the Trophy Garden, so a Max Repel wouldn't have any effect there.