To add more to it, I've come to realize over time that the game makes a point of how naive the Followers tend to be. To a point where they've indirectly created one of the biggest problems for the poor and struggling communities making up the Outer Rings of Vegas - The Fiends.

We know the Followers taught the Great Khans how to make better chems in the hopes that they'd begin developing a medical distribution network. Producing and selling much needed medical supplies for the wasteland in ways that the Followers could not.

Of course the Khans, being a notorious band of theives and murderers, took their newfound knowledge and began a drug distribution network instead - which included supplying the Fiends and feeding their drug habits. Making them even more violent and unpredictable than before.

Not to mention that pretty much everyone seems to be in agreement that their noble people who are trying their best to do good, but their ideals are at odds with the facts of life around them. Which is why they've failed to really grow a significant base of support despite existing for well over a hundred years by the time of New Vegas.

I relate a lot to Beatrix's perspective on them, that being the Ghoul Bodyguard at the Old Mormon Fort - "If they didn't charge for their services I'd think they were crazy, but no, they're just naive. Warms your heart."