Many factors. The education around English isn't really that great, unless they take advance courses. Some students just aren't interested in English, where some schools (not all) don't require mandatory english classes. But mostly its because of the rules (like grammar and such) are so far apart that it's hard to speak Japanese. They have better chance learning Spanish, because some rules are similar and some words are similar. Atleast in my experience, don't quote me on that.

English is far more articulated with our Alphabets (A, B, C, when you say a word, you hear each individual letter) where in Japanese, Alphabets are stuff like Ru, Re, Ko, Ro, To, Aa, e, shi, so, ect ect. Those make up words like Shite (Shi - Te), Ruko (Ru - Ko), Yuki (Yu - Ki), ect ect. Also L, X, and V doesn't exist in Japanese. So you get pronunciations like Rabu (love) or insert funny words basically. So they try to get close to it.

Not to say it's impossible, I know a lot of Japanese that are super fluent in it. Just it's not a major interest in the general public. By the way, not that I know everything of the language. Just as a Spanish speaking person learning Japanese and fluent in English, French, and Dutch, I can see why it's difficult for Japanese people to speak fluent English.