In The Tragedy of Julius Caesar Act II why do the conspirators finally agree not to attack Antony?

Brutus is afraid that if they kill Antony too, that would be a great injustice because Antony was an honorable and loyal man. So, they decide not to attack him.