There are a few episodes where they have conflicts, but none involve physical violence:
* "The Aftermath" (Season 1, Episode 13): This episode shows tension between them as they are both vying for the attention of Caroline.
* "The Election" (Season 1, Episode 20): Charles and Almanzo are on opposing sides of a local election, which causes some friction.
* "The Runaway" (Season 2, Episode 9): Almanzo and Charles have a difference of opinion on how to deal with a runaway boy.
It's possible you're thinking of a different show or a fan fiction story.