I don't really have any suggestions, but I can relate. I grew up in a household in which my parents spoke an Asian language - they didn't know much English. Despite being immersed in a foreign language and even after taking formal courses to learn the language better, I could not understand the language when spoken. My problem wasn't just with spoken foreign languages; I also had problems understanding lectures in English when I was in college, though it wasn't as bad as listening to another language that I was trying to learn. Much later in life, I was diagnosed with Auditory Processing Disorder.

I now know that I can't ever take a class in person if I wanted to go back to college. Fortunately, with the proliferation of online courses provided by edX, Coursera, Udemy, and others, I can learn whatever I want by watching video lectures with the subtitling turned on so that I can follow the lectures.