I've explained what I think below but you're still right: if we assume that it works like a hot poker then it sorta makes more sense to skew damage from all sources to higher values. I just don't assume that damage works that way and I'm used to not letting minutiae drive the mechanics (because D&D is not a physics engine). Also, I see the idea of theatre a bit differently to how I actually suggested.

It is helpful when it comes to rules, to see that the numbers translate most easily to theatre and not the other way around - we get problems with rule interpretation when the theatre obscures its own basis through over interpretation. D&D always falls to pieces under enough scrutiny because it keeps things abstract, but that's the price we pay for a light system of rules.

As the mechanics are the underpinning of how we construct theatre, theatrical sense in D&D is derived as much from interpreting the general patterns in the rules, as anything else. It makes no theatrical sense to reroll the bonus spell damage, once you're used to imagining things unfolding according to the patterns of the general rules - by the same token, RAW you might not expect to roll smite and sneak attack damage twice when you crit just by looking at the first part of the entry for crits, but it's written specifically in the last bit as an addition because of what attack damage means. When rolls stop making sense, that becomes jarring, and so generally I think we tend toward a specific type of mechanical theatre to avoid dissonance. We hand wave this with most things being abstractions. Damage and hit points for example, don't translate simply to wounds, and this makes it easier to tell a story.

To return to the meat of my disagreement, even if hit points weren't abstract, and instead quantified health and wounding in the way you describe, the damage die of the weapon and your relevant modifier don't natively affect the spell or smite damage. Therefore there is very little mechanical-theatre justification for weapon skill and any direct bonuses to weapon damage, affecting the outcome of a spell with the exception of crits.