I have been asking myself the same thing. Am about to buy a Series X next week.
The main thing to consider is that the Series X is currently the most powerful console out there in the Microsoft family (not counting PC's here, that's a whole other story). Yes, you might buy the Series S because, as you say, you play mostly sports games and might not be interested in 4K gaming right now. But what if that changes?
Let's say a year from now you want to play 4K games and have the very best performance while gaming. You won't get that with a Series S. I'm not saying that the Series S is a bad console, but the fact is that it is inferior to the Series X.
If I have to give my own opinion I say go for a Series X. Yes, it is more comprehensive and maybe a little too powerful for the games mentioned but you will be all set for future games and already assured of the best performance instead of regretting not buying a Series X when you had the chance.