I think people focus too much on the word "selection" without considering that the word "natural" means that there's not actually a guiding hand that selects for the best traits. It leads people to say things like "evolution chooses x trait over y trait," but evolution doesn't choose anything. It doesn't have a brain or a plan. It's just a word we came up with to describe natural processes that tend to proliferate the most able members of a species. But that's not always the case. There are outliers.

Like you mentioned, the only thing that really matters is which members of the species are most likely to survive to reproductive age and then actually reproduce. It doesn't necessarily have to be the biggest, strongest, smartest, or fastest members of that species; it could just be the members that have the highest sperm count. Being bigger, stronger, smarter, and faster usually helps to survive and reproduce, but it's not always definitive.

Fish are not what most people would call "smart," and a lot of them are small and weak as well. Yet there are trillions of fish because most of them tend to reproduce extremely readily and have plenty of offspring. If smarts, strength, and size were all that mattered, then sea mammals would have wiped out fish long ago. But species like whales and dolphins, which can kill huge amounts of most other aquatic creatures with ease, reproduce much slower by comparison and are thus vastly outnumbered in the ocean compared to fish.