Attending a church service as they were traditionally held impacts the rights of others.

The vast majority of whom would chose to allow it by attending with them.

People are still allowed to worship and attend church, just not in person

That doesn't really replicate attendance.

they are just banning people from causing harm to the public

Yet they have left daycares and public schools open where children who are incapable of social distancing or properly wearing masks are at far higher risk than mask wearing, socially distanced adults in pews, and have stopped the sale of neither alcohol nor tobacco products despite their direct correlation with increased health risks. Which is where the argument that only preventing worship to preserve life falls apart.

Which brings us back to agreement with my original point, rights of expression and religion have never been absolute.