Well, the last thing you do is start with quadratic equations at age 3. But some age appropriate learning exercises are the way to go. There is a gigantic difference between learning something to understanding, and merely learning to parrot what the teacher says. The former is the key to really utilizing knowledge, the latter is (at best) only useful for passing the next test before it's forgotten. If you're one of those who believe teaching to the test is the smartest way to get kids to succeed, I'm very sorry for you. But if you're really interested in finding out what to do for kids to help them learn for realz...
...Consider this...
Zero is a weird concept: how do you describe nothing to someone? This is where brain development comes into play, because if the part of the brain that deals with abstraction isn't there yet (ever notice a newborn's head is about the size of an apple, while that of an adult is the size of a melon?) then you can "teach" until you're blue in the face, but the lesson will be (at best) parrotted back to you, with no real understanding taking place. That's teaching to the test at its best. Anyone who says "get them going early" is projecting their adult fantasies onto their children. This is unquestionably bad.
Similarly, most algebra equations are reliant on understanding that things can "not" be there, and yet still be represented by a placeholder. This is also a gigantic concept -- very related to the idea of zero. Ever play "peek-a-boo" with an infant? The whole reason why the kid loves that game is because you literally disappeared in the kid's mind. This created a mild amount of angst, but then you reappeared, and the kid is happy because you're back again. You can literally "hide" something under a rug in front of a toddler, and to them, it's gone forever because it's not immediately visible. The part of the brain that deals with remembering something was just there? That's not come online yet. So all the bits of algebra that depend on placeholders just won't work. Those neurons don't exist yet. (Related: why can't you remember what happened to you during infancy? Again, those neurons hadn't formed.)
There are literally stacks of books and academic papers which deal with age appropriate lessons for every subject in school, so I'll leave it to your own research if you need more specifics. But the short of it is that kids need to learn what their brains can handle, and they need not to be pushed into "learning" things if the required parts of their brains don't even exist yet.