They're only unfair in the way everything else is unfair, owing to individuals having different aptitudes and capacities. In sports, bodily traits happen to be important. But so what? Your math example is fitting. Some people have a better knack for numbers than others, which gives them an advantage in math competitions, and on exams and so on. It's not fair. Not at all. Likewise for every domain of human endeavor. Aviation, politics, poetry, fashion design, engineering, music, you name it. Maybe differences in natural ability seem more blatant in sports, but they apply everywhere. Life is not fair. You're singling out sports because they're frivolous? Well, they're not really. They inspire people. Athletes are role models. Sports promote nutrition and fitness and the whole "sports industrial complex" if you will. Think about equipment manufacturers, clothing retailers, all the people employed at stadiums and so on. The economic impact is real. That sector of the economy is no more frivolous than anything else with an entertainment dimension. Is the video game industry invalid in your eyes? Cinema? Are music festivals BS?