I’m an aeronautical S&C engineer and this is spot on, but also worth mentioning that another large contributing factor is the location of the thrust. The further aft the nozzle is, the worse your directional stability is. The X-15 has the nozzle in the aft most location— as is also true with most fighter aircraft. The difference here being that the more thrust you have, the bigger this problem becomes. The X-15 has a literal rocket engine, so a huge amount of thrust in the worst possible location is a nightmare for directional stability. I haven’t done the math, but I would imagine that this is likely probably the main contributing factor to the vertical tail sizing.