First, this is not actually the problem at hand. Yes, this may be how many lay people argue on the internet, but most lay people are pretty uninformed about sports science and sports regulations. The problem we're actually dealing with is the following:

1. There is no real disagreement that eligibility for women's sports (or men's sports, for that matter) can be regulated. Not even all women that are neither trans nor intersex are necessarily eligible to participate in women's sports, after all.
2. There is no rational line of argument that would support the ban of all trans women and trans girls, categorically, without exception, that relies solely on biological sex differentiation and fairness/safety arguments.
3. Where, how, and why to draw a line is a difficult question that is best left to experts. And if you talk to experts, you'll find that this is actually a lot harder than most people believe. (Because most people base their arguments in this area on stereotypes and/or an incomplete understanding of the sports science involved.)

Like first of all gender identity is not a physical characteristic. The current system divides people up by gender as “men” and “women” by using hormone levels, and even decides you should identify as a man if your testosterone is too high (even if you’re a cis-woman). So even if you think “you’re a man if you have a penis” then this system is still wrong.

1. That is not actually how any of this works. That is literally not how any of this works. Hormone levels are only one part of a fairly complex decision tree.
2. This is complex because sex is not naturally binary. We have intersex athletes, trans athletes, and to make matters even more complicated, athletes who are both trans and intersex. To shove all of them in one of two categories requires a complex set of rules.

3. Rules need to not just exist, they also need to be implemented. For example, you don't want to require every female athlete to undergo an invasive and degrading medical exam to determine if they are actually female. Yes, that actually happened for a time:

   "In some cases, this involved what came to be called the “nude parade,” as each woman appeared, underpants down, before a panel of doctors; in others, it involved women’s lying on their backs and pulling their knees to their chest for closer inspection."

   The existence of trans athletes who are also intersex complicate matters even more. There are trans women with XX chromosomes, trans women who have a uterus, and so forth. Plus, for some women it is basically impossible to tell whether they're trans or intersex with just medical tests.

   Thus, regulations are also designed with an eye towards practicability.

4. Testosterone levels are already tested in elite sports as part of anti-doping regulations, so it's fairly convenient to use this as a starting point to deal with edge cases.

Maybe a genetic spread would be a better indicator, maybe bone density, or height but what disadvantages are we actually accounting for?

Virtually all the sex differentiation that is relevant for sports is the result of hormonal action. Where things get difficult (and where we're having a huge chunk of the argument) is that both previous and current hormone activity matter.

The other thing is that contrary to popular belief, sports often aren't actually segregated by biological sex, but by weird mixtures of biological and social factors. This is because historically, sports were segregated for social reasons, with fairness arguments based on biological differences being a post-hoc justification and often difficult to retrofit.

Look at the World Athletics rules, for example, which define eligibility for the male and female categories (C2.1, Technical Rules):

"3.4 An athlete shall be eligible to compete in men's (or universal) competition if they were either born and, throughout their life, have always been recognised as a male or comply with the applicable Regulations issued pursuant to Rule 3.6.1 of the Technical Rules and are eligible to compete under the Rules and Regulations.

"3.5 An athlete shall be eligible to compete in women's (or universal) competition if they were either born and, throughout their life, have always been recognised as a female or comply with the applicable Regulations issued pursuant to Rule 3.6.2 of the Technical Rules and are eligible to compete under the Rules and Regulations."

(Rules 3.6.1 and 3.6.2 refer to transgender athletes, and there are further rules for intersex athletes that apply only to certain events and intersex conditions.)

What you're seeing is not only that this is a mix of social and biological criteria, but that even the biological criteria ("born male/female") are subjective. Consider, for example, a country such as China, which is not exactly reluctant to literally bioengineer kids for their athletic potential to proactively assign intersex athletes as female at birth if they think they might become successful athletes in order to leverage their potential.