The effects definitely haven't "gone away" but I don't feel like you should feel guilty either. The systemic abuse of an entire race isn't your fault and may not be the fault of your ancestors but the reality of the situation is that it happened and the ancestors of those people still feel those reprocusssions. Even today, capitalism kinda fucks the poor and undereducated. When multiple generations are forced into slavery and restricted from education it obviously has a lasting effect. Even if modern society was completely free of racism (it's not) those with enslaved ancestors are working with a disadvantage compared to those that weren't.

I am white as can be but my parents didn't even graduate high school. I managed to get a college degree but had to join the military to do so. Beyond that my money management skills aren't great and my parents have a lot to do with that. My failures and successes are all my own but my upbringing added a little extra layer of challenge to getting where I am today. My point is we aren't ever a clean slate. Your parents, their parents, their parents parents, have all effected where you are I. Your life right now. Now just imagine that your entire lineage was refused an education and living in poverty was the norm. The chances of you rising above that would be tough and then add the fact the some folks don't like you cause of your skin tone. It may not be your fault but you need to realize that there are large populations of society that have been trying to make it by day to day just like you have but have the added handicap of not having an educated or prosperous lineage. That kinda of disenfranchisement lasts a lot longer then a lot of people realize.