My father-in-law is the neighbor from hell.
He's the stereotypical balding, fat, uneducated slob that spends his days sitting outside wearing t-shirts and shorts that are about 7 sizes too small, with his gut hanging over the top.
He's a junker who allows his junk collection to clutter up his yard (which for the record is not fenced in), and he keeps an old, beat-up pickup truck and trailer out on the street. Since his junk collection is not hazardous, he's technically violating no law/ordinance/code.
His house is basically a run-down shack in the middle of a neighborhood full of properties that should be valued far higher. But their property values have declined because of the condition of his house and his ever-growing pile of junk. Prospective buyers who are going to spend $250k+ for a house (the going rate for much of the area) don't want to live next to a house that looks like Fred Flintstone's (Yes, it's made of stone and everything), nor do they want to live next to a junkyard where some fat slob sits outside and does nothing all day.
Neighbors have sought legal action, but since he's not breaking any laws/ordinances/codes, there's nothing they can do but watch their property values fall. He's actually told people that it's his property and they can mind their own business.
Is he 100% within his rights? Absolutely. He can spend all day flipping them the bird, and there's nothing they can do about it. But he's the most hated man in the neighborhood, and the reason that HOAs exist are because of people like him. Of course, the problem is that HOAs end up going to the other end of the spectrum and end up meddling in affairs they have no business in.
It's basically a "pick your poison" situation. Join an HOA and be forced to pay dues to a bunch of little power-tripping dictators who probably haven't gotten laid since 1972, or risk having someone like my father-in-law for a neighbor and watch property values fall.
(Personally, I'd never join an HOA. While I strongly disagree with my FIL's handling of his affairs, I strongly agree with his philosophy of my property being my business. The major difference between us is that I'm not a slob, and I actually do take how my actions affect my neighbors into consideration, and make my decisions accordingly.)