" When he starts talking to my face, his eyes just darts for a second and then he immediately bends down. " shyness, maybe? Many people struggle with eye contact.

I dont think his intentions are malicious or perverted. It just happens, as a guy you have to get used to (especially men who have never been around women who wear modern clothes like tshirts etc.) women in clothing that is not super baggy. If you want to talk about it with him, be gentle with it and don't do it in public. If he tries, he will succeed. Give him some time after talking to him about it.

Perverted folk often show their perverseness in various ways, not just one particular action repeated over time.