Okay, that makes this whole thing a whole lot scarier - he thought you could 'take the punches'?!?!?! WHAT???

This right here OP is why you should NEVER EVER go back to him. He KNOWS exactly what he is doing and he is manipulating you. First he paints you as crazy and now he can recall EVERY SINGLE NASTY THING HE HAS SAID AND DONE??? That alone should prove to you that he knew EXACTLY what he was doing and did it ON PURPOSE!!!

He is scary as shit and you need to go complete No Contact with him because the longer you continue to talk to him the more he manipulates you and makes you question yourself.

CUT OFF CONTACT!! Block him on your phone or better yet get a new one. Set up a whole new email address and delete the old one. Change all of your electronic user names and passwords. Either get a new phone or do a factory reset on the one you have. Take your car to a mechanic to make sure it is not being tracked. Make sure you are never home without your brother! If he can't be home when you are - go to a friend's house. STOP LETTING HIM IN THE HOUSE WHEN HE JUST SHOWS UP!!! You have already taught him that he just has to show up and you will let him in. STOP THAT.

You need to take this seriously OP. That 'take the punches' statement is beyond chilling and you need to stay away from him. You talking to him keeps his hopes up and keeps him manipulating you.