The short answer is no. It is not going to break your game. Furthermore, a religion that has decent beliefs like Feed The World or Choral Music and is spread to your cities would be immensely beneficial to your civilization and development.
However, the game is harder without it. Religion is the strongest enhancer for any style of victory and focusing on it early can pay off high dividends.
For example, take Cross-Cultural Dialogue or World Church beliefs that can add +1 science or culture for every 4 followers. That means, in the early game you can generate culture/science yields just by spreading religion and growing your own cities. Districts are not even needed.
There is also founder beliefs to choose from. The most straight forward, and universally strongest is Work Ethic. It adds production based off the adjacency bonus of a holy site. Naturally, this one would go well with Dance of Aurora, Desert Folklore or Sacred Paths pantheons. And then add Scripture government policy to double it. This is by far the strongest combo to ramp up production early on.
Also remember, that you can buy Great People of all types with faith. Faith has a higher value than gold when making these purchases, so if you can generate 200-300 faith per turn then buying a great person within 10-15 turns is possible. There are so many other options that you can trade faith for, that it makes sense to make Holy Sites a priority district early on.