Do you think there is some special vulnerability to problem #2 presented by this kind of hidden magic system? For me, it's especially a problem because the motive behind this in the first place has a lot to do with being in the game world, but I was wondering if you had something else in mind.

I don't have much else in mind in particular. It's just that I've seen some people go crazy at the mention of "my character is aware of magic points beyond just feeling tired." So I think you would do well to avoid this outcome by doing some careful design so that the meta-elements and in-game elements would not end up with something along those lines that would make someone at the table squirm. This sort of thing very inherently blurs these lines. Another thing to keep in mind is that you're going to need to use some kind of logic as to how it works, which will also help with that, and will also help prevent the use of "magic" as a button for "whatever the hell I want" sort of thing.

First, the PCs could just be heroic figures with a high tolerance for magical injury. Second, the players could just be advised to tread gingerly in their magical research - which of course would require some kind of codification of what treading gingerly or not would look like.

"High tolerance for magical injury" doesn't quite do enough. High resistance to curses, and some way to contain something that might go out of control, would probably be necessary. UNLESS you want really nasty surprises to be a thing in there, though IMO players should be warned that this is a real possibility. Personally I see it as being possible to essentially use extremely tiny amounts of substances and magical power to produce tiny effects, which if reproduced on a large scale become actual magical weapons or utilities (e.g. healing). For instance, an experiment involving channeling inner magical power and a couple of plants results in petrifying the skin on a fingertip for a few moments, but if you reproduce it on a larger scale you'd be able to have potion you can throw that may petrify for hours.

Treading gingerly would require a lot of careful consideration as to how that's going to be. You're going to need to give them enough to work with where they don't feel like they're face-mashing into a land mine to see if it explodes. I can see this pretty easily resulting in the stupidest guy in the group essentially doing this, having it blow up, and everyone else deciding this magic stuff isn't for them and deciding they would rather not take their chances with a ticking time bomb.

Maybe each character would have in addition to their stats a pool called "reserve" (or whatever). ... They could spend the points whenever they wanted to and quickly discover that it helps you out sometimes and somehow, but they don't know what happens when you hit zero or how to recharge it.

In terms of "how to recharge it," I would resist that impulse, because that would encourage hoarding it. Even the risk-takers might do that, because they would be concerned at experimenting with it and dumping it out in experiments that ultimately go nowhere, and they can instead use it as a big, flashy, last-resort option to use all at once when it looks like there's no way out and/or they're confronting what seems to be the final opponent. If you do this, they should figure out how to recharge it very early on. In fact, an abundance of magical power and an ease of recharging it would probably be a very good idea so they have stuff to work with in the experimentation.

I imagine it might be good to have them run into a wizard that explains the basics early on, although said wizard may well afterwards say, "Well, now that you know the basics, obviously ya don't need ME anymore!" kerpoof That would cover the bare minimum they'd need to know, including how to recharge it. Or it's possible that it's used for something else - perhaps some form of weak magical power is already commonplace, which would familiarize the characters with the basics, but they may have some sort of intelligence or natural power that goes above and beyond what most can do, at least in the realm of experimentation. This might also be a sub-goal for the campaign - bring a new form of magic into existence.