NAH it's definitely frustrating to be misheard when you feel you are just speaking normally, so I get why he was upset. However, if you genuinely couldn't understand what he was saying, what were you supposed to do? Yes, I can see how it would have come across as rude, but you have to find some solution if you are wasting both of your time repeating over and over while gaining no traction. It's not viable to always ask your coworker to translate.

I do have to ask, are you the only one who has trouble understanding him? If so, that might be exacerbating the situation as he might feel you are trying to underhandedly insult his accent and get him to "talk normal" in some way. I would try and sit down with him and explain that this isn't a problem unique to him, you frequently have trouble with needing people to repeat themselves and you'd like to work with him on a solution that makes things go smoothly for both of you.