If you don't mind, I'll try to mansplain why your friend might have reacted that way, assuming that your friend has a sizable history with you and knows that there is no romantic interest.

You complimented him, activating the script that compliments = interest. This ran up against him knowing that you DON'T have interest in him, based on a months/yearslong dataset.

So he's processing that conflict, and at the same time running a script to not express reciprocal interest in you. (Negative outcome of "stumbling, defensive uncomfortable reply" = a lot less than the negative outcome of "be perceived as making advances on you." I don't think you're going to stop being friends with him because he reacted with extreme discomfort. If he reacted by complimenting you, you might see that as unwelcome flirting, and you'd have good cause to end the friendship.)

BTW, I'm not asking you to feel bad for us--most of this cycle of compliment-desert > compliment over-reaction is both our fault and our problem, partially because of homophobia and partially because of just conformity. We don't compliment each other, initially because it would be gay, later because it would stand out.