If you want the person to visit the dog, set-up a specific schedule on specific set days and leave the dog in the garden for 15 minutes. They either pick the dog up in that time or they don't. It doesn't have to be every day but it can be 3 days a week. If they value time with the dog, they'll meet this. It's not on you to be an open-door policy for someone you're doing a favor.

If you don't want them to visit at all, or they refuse to keep to a pre-determined schedule and expect you to accommodate them, tell them to find a new place to keep the dog.

​

YTA for being a doormat, but otherwise, not. However, I'm going with YTA for simply because you didn't have to be in this position, you're allowing yourself to be placed their by your own weakness.