Using magic makes the user less magically potent.

"Mana" in my primary setting (which isn't called that but there are a billion names for it and I'm going for clarity here) represents both the fuel and potency of a magic user. The more mana you have, the more powerful your magic is capable of being. There is also no "cap" to how much mana someone can obtain, but it's a difficult and time consuming process. As a result of this magic users are very cautious and conservative with their power, because every use of it literally weakens them. A good magic user is one who can produce a desired effect with the most subtle magic possible, so as to preserve their power.

Also if a meaningful amount of mana get loose with no controlling mind a bunch of crazy stuff is going to happen, and it's not going to be cute shenanigans it's going to be straight up body horror if you're lucky.