There's no way to predict which countries could have been great. A mere 200 years ago, Finland and Norway were two cold and poor regions in northern Europe. Today these nations are among the most developed in the world. We cannot know how history would have turned out had things not developed the way they did.
The negative effects of colonialism are undeniable. An estimated 11 million people were displaced from Africa over the Atlantic. Millions more were killed during the conquest of the Americas. European colonizers destabilized entire regions by supporting and heavily arming particular groups/tribes/kingdoms. And while colonies may have been costly, it was an investment with an expectation of return – in other words – designed to bring colonizers more profit, not help the local population. Asian countries that got colonized never managed to "brush it off pretty fast". Quite the contrary, former Asian colonies are severely underdeveloped down to this day. Europe didn't do anyone a favor by colonizing the world.
Believing that physics, math, literature, and other forms of culture or knowledge are exclusive to Europe is minimally naive and maximally chauvinistic. These existed in captured territories. If anything, Europe merely had better opportunities (as in wealth) to invest in research and development. This does no mean it was exclusive to the continent.
Europe may indeed not be in a good situation, but the mainstream theory (internationally at least) is that if people are to enjoy the benefits of their forefathers' actions, they should bear the burden as well. Europeans in particular believe and acknowledge that:
All human beings have a set of unalienable rights and that Europe must uphold this standard.
Europe had an massive impact on certain parts of the globe and are partially responsible for the current state of multiple nations.
As a result of these beliefs, the current consensus is that Europeans should (1) help out their fellow man and (2) help those suffering (either partially or completely) from Europe's actions.