If my eyes were darting around the room, I reckon it was because I was uncomfortable being stared at like that.

This makes sense. The guy likely knows he has an intense gaze and that might make him hyperaware of what other people are looking at. I have a lifelong friend who’s genuinely kind, but has been told he has “a Ted Bundy stare” and “crazy eyes” because he doesn’t blink often and stares at people. It’s genetic in that his brother does the same thing & seems to enjoy that it makes people uncomfortable, whereas my friend will actively control the staring when he first meets someone socially or at work and tell friends he knows he has intense eyes and isn’t trying to be weird.