I have the Harmony 650, here is what you can do:

• Add devices via their computer software, just enter model numbers.
  
• Dedicated activity button: You program this in the setup. For instance, I programmed my "Watch TV" button to power on my tv, receiver, and cable box; the receiver also switches the input to HDMI port for the cable box, the remote then acts as the cable box remote, and the digital screen as controls for the tv (picture modes) and receiver options (audio mode). For my Apple TV, I used the "Listen to Music" button, and the remote does what it should and now acts as a remote for the Apple TV.
  
• Off button: turns off any devices it knows is on.
  

If the HDMI switch has IR control, it should work. Check the Harmony's device compatibility website to see if it's supported.

For the Xbox One, it can power it on/off. Not sure if navigational buttons work, but text entry would need to be done on the Harmony Smart Keyboard (which I guess means you need the hub) and the USB receiver plugged into the Xbox One. So, unless you want to do that,!you would still need to use the controller.