I hear you. I grew up in a family of mathematicians who were convinced that the hospital had given them the wrong child. I will always remember the day my dad said “how are you going to feed your family? Draw them a steak?”
Well, turns out that yes, i can draw a steak and it feeds my family
In the end though, I chose this path, partially because I don't like being told I can’t pursue what I want with my life, and second, a burning desire to prove every one of the nay-sayers wrong. The othr half was because my art was such a powerful and zen-like escape from the world around me, that I loved living inside my paintings as I created them.
You can do it, but it all depends on how resilient and determined you are.