Now if your brother needed a kidney and you were the only one able to give one and you refused, I'd call that maybe cowardly or failure to fulfill your duty to your fellow man

I don't see how this isn't directly comparable to abortion. By having the fetus inside your body, you are the only one that can donate your body to the wellbeing of that fetus - in the same way that you're the only one able to give the kidney.

The only difference is that the "Action" in that case is to give the lifegiving kidney while "Inaction" is to allow your brother to die, while in a pregnancy, they're inverted. "Action" is the abortion and "Inaction" is letting the pregnancy run to term. And at that point, we're running into trolley problem territory. If the only determiner of what actions to take is the goodness they give to others, action and inaction should be no different, but I think ultimately we see the consequences of Action as being more meaningful than consequences of Inaction. We are more worthy of blame for bad effects of Action than Inaction.