I have an X and a 4k tv but yesterday I took it over to my buddy's place and hooked it up to his 2006ish plasma and it still looked pretty good. The difference in resolution is clear but we were playing Forza and he said he was noticing a lot more detail than usual.

I think the biggest thing is that the X runs most games at a higher frame rate and a lot of games have an option for 1080p performance mode versus 4k graphics mode. It's left up to developer as to what they include but if you care about smoother frame rates and in a lot of cases better lighting/graphics then I think it's worth it.

Plus, 4k tv's are getting pretty cheap and if you're the type of person who can justify buying an X you can probably get $275 together for a cheap 43" tcl.