You mentioned "unexpected leave" which likely means that you didn't tell him before leaving and he didn't find out so to him it probably just seems like you never took him up on coffee because you weren't interested.

Next time he makes eye contact, smile and wave to either get him to approach you or you approach him. Explain what happened and take it from there.

Personally, I don't think two colleagues grabbing coffee is a date so you're not breaking any rules or being inappropriate. Also, make sure your company doesn't have policies against dating a co-worker for later in case your "coffee catch-up" goes well and you do want to go on a date. Good luck!