The Series X has more CUs which has been found to give better performance, high clock speeds has diminishing returns.

There are diminishing returns for both. More CUs doesnt just mean more perfomance because a lot of games arent going to fully utilize all 52 CUs. Processing power in general has diminishing returns, so there isnt a 1:1 ration between power difference and performance difference.

The real answer to how the differences between the consoles will manifest when it comes to actual games isnt something anyone can answer right now. It's most likely that third party games will be close enough that people will have to rely on things like digital foundry to point out the differences for them. First party titles will be the ones that really take full advantage of the differences, so we can only wait and see. And as you said, we wont really be able to truly see this for probably a couple years after the systems are released.