Most of our college and university education in India is in English, which means that if a person wants to pursue higher studies, they need to know English well. This is not the case other countries such as China, Japan, South American or other European countries, etc. Their education is usually in their own native languages, including higher education. They learn English as a second language. The language in the workplace is also largely their native tongue. Thus a person may be well off and highly educated, but not be very comfortable in English, and this would not be unusual.

Meanwhile in India, English is almost the sole medium for higher education, our laws are in English, and most corporate offices and workplaces use English to communicate. We use the language a lot more than other non-English speaking nations. Thus a person who speaks broken English is probably one who has not gone to an English medium school - which is a mark of class, they would find higher education more difficult to access, and would have trouble in the workplace.

Because Indians associate English with education and socio-economic class, they tend to look down upon those who don't speak the language well.