There have been different accounts over the years about the nature of this difference. Some sources state that Potter felt he wasn't given enough to do in the show's direction, while others say he disagreed with the darker and more dramatic turn the show took in the second season.
Ultimately, the reason for Potter's departure remains somewhat unclear. However, it is clear that he and Davies had a disagreement that led to Potter's decision to leave the show.