Open racial prejudice is simply no longer as prevalent as it used to be. It's still there, but it is not nearly as socially accepted as it used to be. But the institutional racism still lingers: The effects that the segregation had on black people is still apparent, for example if you look at the financial entry barrier for higher education and the resulting effect on demographics at universities. Things like that are a far bigger issue than racial prejudice today, people talking about racism of any form are more likely to talk about institutional racism. Hence, the term became much more closer associated with it.

This is simply the way language evolves. There are lots of words which have more or less changed their meaning completely, or even mean entirely different things if you go to other nations. Just as an example: When people from the 19th century used the terms "gay" or "queer", they were probably not talking about homosexual people. Now they are almost exclusively used in that context.