What went wrong with transitional goal of the Wii U?

To put it simply, the gamepad was an expensive piece of hardware but simultaneously a shitty piece of hardware. And I think that's the main area where it failed.

The screen scratched easily and was a very poor quality, especially in the modern era of smart phones where almost every single person has an example of a much better screen in their pocket at all times.
The audio quality wasn't great either.

Even if you pretend it had a good shape, the actual controller itself still wasn't that great to hold. It was very light, which made it feel very fragile regardless of whether or not it actually was. Meanwhile most actual controller manufactures are doing what they can to make their controllers feel heavier and more durable.

The bonus screen wasn't very useful either. It was pretty much used for the same shit the DS extra screen was used for, except the DS screen is touching its other screen and the Wii U extra screen typically requires you to actually away from your main screen.

There's like only a couple games out there that make good use of the screen to begin with.
(No, the remade Zelda games that already worked perfectly without the touchpad don't count as a "good use")

If you had taken away the gamepad, you could have built a better and more powerful device at the same price.

How might the Switch cover the ground where the Wii U fell short?

I think its already done a good enough job. For starters, it seems far more worth the price than the Wii U was just off the hardware alone.
The branding is also significantly better.
The games coming out are better. (Hell, half of them are Wii U games rehashed and upgraded for the Switch)

How can the other big players respond to this as the next generation of consoles approaches?

I don't really think they need to. For better and worse, Nintendo has pretty much distanced itself from the competition's methods in almost every single way.

There's not a whole lot of room for competition because of how distinct the Switch is from its competitors.

Triple A games and triple A devs go to the PS4 and the Xbone.
Nintendo games go on the switch.
If you want decent online services, you avoid the Switch.
If you want your favorite PC indie titles to be somewhat mobile, you got the Switch for that.
If you want a decent multimedia device for your home theater / living room / bedroom / etc, the other consoles can do that.