Seriously -- you've been too accommodating to this whack-job and he sees it as the weakness that it is. (Not your fault, women are socialized from day 1 to be accommodating.)
Your own senses are telling you that this guy is dangerous. He can hurt you--easily.
So take some of the advice you've seen here, and make sure your family is all on board with it.
* Read the book "The Gift of Fear".
* Start thinking proactively about your safety and security. Walk around your house & imagine you want to break in. What are the weak points? Are your windows and doors secure? Now get into creepy neighbor's head and think of opportunities that this man has had to watch you and ways that he can attack you.
* Change your phone number & stop giving it out so freely.
* Find out from other neighbors if they know anything about this guy and let them know what he's been doing to you--and don't downplay it.
* Start taking a self-defense or martial arts class.
* Buy some mace & keep it velcroed to your hand whenever you step out of your locked house. Let him see that you have the mace.
* You and your family should get NRA stickers for your cars so he'll think you've got guns. (Consider taking firearms classes and getting a gun permit.)
I really do like the suggestion of thinking like an apex predator. I don't agree with the rest of that comment -- like I don't think you should act all crazy with the guy. The less you say, the better because it gives him less to obsess about. But I do like the idea of putting yourself in the mindset of being in control. You've got to get control of this situation or you're merely human prey. You've got to become aggressive about your own survival or you'll end up becoming another cautionary tale where an innocent woman is tortured, raped and killed by a creepy guy.
So, figure out what his deal is. Do some internet sleuthing. Look up his name on your state's judicial website to see what sort of criminal offenses he may have been convicted of. Look at his facebook friends. What does he comment on? Where does he work? When does he work? What are his habits? Remember, knowledge is power--so get powerful.
And finally, stop being so nice. You don't have to be nasty or crazy with him. But you do have to set boundaries--so you have to be firm, and you do have to be in control. Somehow you must find the will and the words to tell him to leave you alone. And, you should get a man to tell him to back off too because unfortunately, men like this don't listen to women, but often they will listen to men.
Try this: Have your brother ready to back you up and come out as reinforcement. When you go outside and he comes over, you need to immediately tell him "I want to be alone". He probably won't leave, so turn around & go inside. He'll probably keep talking. Tell him "I want my privacy, leave me alone." If he tries to come in with you say "you are not invited, I want to be alone." Your brother should come out by now and tell him "Leave her alone. Enough is enough. You need to go back to your own house and leave her alone."
That's the sort of thing that will need to happen. It will be awkward and it will take some effort. But it has to happen. He's already stolen your privacy and your time. You can't even enjoy your own front yard and you no longer feel safe at your own home. Your time is not his to take. You are not his for the taking.
Good luck, get strong, and be safe.