I am an engineering and physics student who is going into a PhD, and I did amateur game programming in high school and at college. That doesn't exactly make me an expert to give you advice, but I think I can help. (If I am horribly wrong, send me to downvote hell.)

I'd say your biggest deficiency would in software engineering if you've never created something complex. The examples of things you have done require an advanced understanding of physics/maths, but they likely weren't complex in terms of software engineering. (Have you ever written something bigger than 1k lines? 10k? 100k?) You really should work on that: Learn about design patterns, advanced C++, algorithms and data structures, etc.

While you are doing that, start making games. You'll soon realize that even the simplest games have lots of little parts that you never paid attention to. Make something like Tetris first, but try to make it complete. It should feel like a full game even if it is crap. (Chances are, it will be. Don't be discouraged.) Then move onto more and more complicated stuff. At some point, you should be able to create fairly complex 2D games and somewhat simple 3D games by yourself.

As you progress, you may try to find your niche in more mathematically involved aspects of game development, such as physics engine development. Your background will allow you to blaze through all the maths, but keep in mind that what you'll be doing is different from what you have done in important ways. In scientific computing, accuracy takes precedence over the speed of execution; this is reversed in game development. You should always keep this in mind. If something is accurate enough not to affect gameplay, then it probably doesn't require a more intricate solution.