Semi hijacking your comment because it's high up, but I feel OP and other should know...

Common medical consensus is that by usually around age 3 a child has figured out gender and know what they feel themselves to be.

Increasingly across the United States, doctors and parents of transgender children are embracing their identity as soon it starts becoming obvious, sometimes around age 3. Many say they are finding much greater chances of happiness and well-being when children are nurtured in their new gender identity at such a young age. Although there is not a consensus on the issue, some clinicians who work with transgender children have concluded that when children persistently identify as the nonconforming gender, the best course is to socially transition, or live as the other gender, even at age 3.

OP's son doesn't sound like he identifies as a female, and is tired of being asked about it. Doing boy or girl things doesn't make you transgender, it's just something you know inside. I think it's similar to asking if you're a good or bad person. You don't have to take some test or reason it out, you just know. Most of us just know that we're a good person because that's how we feel inside, no other reason needed.

Advice to OP: I have close friends and family that are trans and married to a doctor and live with another doctor. Go to doctor appointments with him if you can, and at some point you should be able to talk to the doctor privately and tell him your concerns and also the history of him going to multiple doctors. There should be records from all of the previous doctors he can request to confirm what you're saying. I would also do some research into children who identify as transgender so that you can explain to your wife why you don't think he is, and that him wearing dresses or whatever is just him being a weirdo kid that's normal, not transgender. If you have no idea where to look, I can ask/look up in medical books for more info if you want.