This is one of the issues, you may think that you can do it, but if the only math you've experienced is in american high schools and colleges (even through calculus, differential equations and most linear algebra courses), then you haven't actually done any math. You are good at regurgitating memorized steps on specific cues. Math, however, is creative problem solving, something not expected in most math curricula. It could be shown and emphasized at a very early stage, but it's hard to put on a standardized test so it isn't.

I bet you might be able to "do" music, if all it was was reproducing scales, chords and keys on a piece of paper. If this is all you do, but you never hear a single song, then you might say that you can do must, but you just don't like it. Of course you don't like it, you think mindless regurgitation of symbols onto paper is music, you've never been given the opportunity to actually experience music! This is what our education system has done to math. You just bullshit your way through, never actually experiencing math, and you come out thinking that you are familiar with it and just don't like it.

People are being robbed of the opportunity to experience math, and this makes tons of people thinking that they don't like it or aren't "math people".