I don't think his response was right. Since you guys were getting sexual I don't think it was illogical of him to think that the grinding would've led to something more since you were reciprocating, but at the moment I put my hand somewhere and you removed it (no matter where it was) I would've checked with you to see exactly where we're at. Once I corrected my misunderstanding I'd stop doing anything sexual at all and continue on with whatever.

If you made it abundantly clear that you weren't interested, then at the very least he was trying to tease open your boundaries which is a bit alarming. If this was said to me, I would not in any way try to test it or even give the appearance trying to test your boundaries. Also, while it's easy in hindsight to say, I would've taken him groping you or grinding against you as a boundary test and it would've been best to shut things down sternly at this point.

As they say, you give them an inch and they take a mile. Something to keep in mind with any boundary you set with people (sexual or otherwise). Shut it down immediately.