Here's how Bilbo figures out they're trolls:
* Their voices: The trolls' voices are described as "deep and gruff" and "hoarse." This gives Bilbo a clue that they might not be ordinary humans.
* Their conversation: The trolls start talking about eating the dwarves, and they describe how they like to cook and eat humans. This is obviously a disturbing topic and makes it clear they aren't friendly.
* Their appearance: When the sun rises, Bilbo sees their true form. They are huge and ugly, with stone-like skin, long claws, and large teeth. This leaves no doubt that they are trolls.
It's important to remember that Bilbo is initially very scared and confused when he first meets the trolls. It takes a bit of time and observation for him to understand their true nature.