So if her semblance was always on, it should have tanked that attack.

You seem to be misunderstanding what her abilities actually are. Assuming by "tank" you mean "withstand", she doesn't really tank attacks any more than anyone else with aura does, while she's absorbing. She barely withstood the Mech at all at the beginning, she got pummeled to hell! It was only after activating the other part of her semblance, and getting that power boost, that she caught one of the Paladin's punches. We know this, because there are visual cues when she unleashes all that energy she takes in, and they were present when she caught/"tanked" the punch.

As for Neo, she absolutely did not "one-shot" Yang. They had an actual fight. Neo got several hits in. That said, Yang wasn't ready for her, regardless of whether it was a one-shot or not, but as I hope I've explained, that doesn't matter, as she doesn't need to be ready to absorb attacks. The only reason she didn't get stronger after fighting Neo was because she was out like a light. We saw this in the episode. Had she been conscious, like she was when she fought the paladin, she could have been able to activate the second half of her semblance. Hence: Passive absorb, Active release. As has been demonstrated, she needs to consciously unleash that power, but there's no indication that she doesn't passively absorb it.

Seriously, This was explained in the show, both visually, and through dialog. It's one of the most infamously expository scenes IN the show. Ruby flat out says each hit Yang takes gives her power that she can use. each. Literally synonymous with "every one". Not "the ones she tanks" or "the ones she's ready for", each. We've been told how it works, we've watched it work precisely the way it was explained to us, and it's directly based on a PASSIVE ability from another show! What more do you need?