What you're suggesting is that brutality can somehow scale up the base damage from which a poison is derived. It can't, because if it did that for a pure phys skill and bleed, it would be double dipping.

"more physical damage" scales all of your poison ailment damage that originated from physical damage.

Where do they say this? The thing I just quoted from Mark directly states that a poison cannot know what it was. And the manifesto says:

A melee physical based attack build using the poison gem will now no longer have their increased melee physical damage bonuses applied to the poison, meaning the poison will be doing significantly less damage. Does this mean that trying to use poison on any build invested in physical damage is not worth the investment? Technically, those modifiers never applied directly to poison, they only applied to the hit. This then had an effect on the base damage of the poison because that was the final hit damage. Those modifiers will still apply only to the hit, as before, but since the poison is no longer based on the hit's final damage, they won't change the poison's base damage any more.