I agree with a lot of your opinions but it also seems you are a tad over protective, reason they hope you are not around so they can see the kid. Sorry, I'm not going to tell you what you want to hear (100% agreement) but I think you might be a touch over sensitive, but it seems like some of this is legit too. It's hard. Good luck, just try to analyze your behaviour too.
My mil has a tendency to reach for the child without asking. It pissed me off to where I almost bit her hand one time. Then I realized it's just her style and I stopped being hostile towards her and things improved. Now she asks. It took a lot of effort on my part.