These patterns are actually fractal visualizations, created by a computer program. The exact nature of the fractal patterns changes from episode to episode, but they're always abstract, swirling, and ever-changing, contributing to the show's eerie and mysterious tone.
They are not actually a specific "thing" but rather a visual representation of complex mathematical patterns that evoke a sense of the unknown and the vastness of the paranormal.