I'm not of the "magic only as a last resort" philosophy, but consider this.

If you can't dig deep within yourself to set boundaries, address his problems, try and let him correct these issues and then put your foot down if he crosses those boundaries, how effective do you believe a spell or ritual will be?

I'm not saying it's unethical, but generally the will and intent of the caster is how effective the outcome will be. Also, being inexperienced and not being confident with it can cause outcomes you don't want or didn't expect.

Alternatively, you're probably not the only one who feels the way about him. Might be good to confide in another relative and figure a way to go about it.