I already got several loans to pay him almost $5,000 back for gifts

You do NOT have to pay people back for gifts. That's why they are called gifts. If you had to pay people back they would be called "things you decided to buy for yourself."

he even told me that I better be ready to get another loan.

No you do not do this.

I just want to move on with my life

Then block him entirely, on your phone and on social media. Ignore him completely. Block his email.

does he have a strong case against me?

Not from anything you're written. If he actually sues you (99% chance he will not) the judge will almost certainly rule in your favor. Only you can get him out of your life. Most of this is not a legal issue and better suited for /r/relationships ... good luck!