I followed (and currently trying to get back on track), swimming 3x lifting 4x each 1hr routine. Diet definitely did the heavy-hitting. Eating 500 calories less is easier than burning 500 calories more, especially as a fat person who is (normally) unmotivated by the idea of going to the gym and is already eating a lot more than needed and is (most likely) ashamed/embarrassed/not-confident going to the gym. You can't replace a good diet with workouts, any trainer will tell you that. Hell, anyone who lost weight or is trying to lose weight will tell you that.
3x swimming 4x working out with rest days is a huge time and energy commitment. Eating 500 calories less is significantly easier, and you're likely going to end up eating the right shit because you have a limited number of calories available.
Also, note how you said "for most athletic people," and "skinny fats." You're describing slightly overweight people. Not fat, not obese people. CeRq was obese when he started, as are many people.