The problem with the Followers is that if they didn't have bodyguards, or live in fortified communities, they'd all get themselves killed or worse.
I think Beatrix, the ghoul bodyguard they have at the Old Mormon Fort, says it best:
"If they didn't charge for their services, I'd think they were crazy. But no, they're just naive. Warms your heart."
In other words, I think everyone is so taken in by the Followers and their flowery worldview because there's a profound innocence to it. Refusing to lose hope that a better world is possible, no matter how bad things get.
It's noble of them to keep perservering despite everything, but they have almost nothing to show for it more than a hundred years on from their establishment in 2161.
In fact, in some ways that the game makes clear, they've actually inadvertently made a few things worse because of their naive desire to help everyone (e.g. see them helping the Khans learn how to make better chems in the hopes they'd start making/distributing medications throughout the Mojave).