First, and foremost, if you haven't already please tell your SO about this. I'm sure someone in a four year committed relationship would have already, but just in case you haven't, tell your SO.
I think there's a few options here; report him to the person above him, ignore him, and send an email.
If I were you I would probably send an email, but I think all the options are valid. I would say basically what you said here
You did not flirt with him at his party
You're not, nor have you ever been interested
You know he's married with kids
You're in a committed relationship
I'd also add something like "at this point I find your interactions with me creepy. It's disgusting you did this while married. Your messages uninvited. Do not message me again or I will report you." This way if he replies again, you have proof his advances are unwanted.
I'm sorry this is happening to you, and I hope things work out. Keep us posted.