In my opinion you really didn't do anything wrong. So you didn't say anything when he slapped your ass (you don't have to tell you SO that you were mildly attracted to him) it's human nature to be attracted to others even in a relationship. But this one might be a tough one for him to handle.

As far as the semi-accidental date. You didn't think it was a date, Steve did. You agreed to have coffee with a co-worker mainly because of your stress of moving/family, etc and he took it way to far and you didn't reciprocate. No harm done and your SO can't be mad at you for that.

You need to tell him that going through these odd events made you realize how much you truly love him and that your actually less hesitant and excited for the move vs keeping it in your head and having a guilty feeling. Sounds like if you don't it'll rip you apart from the inside!