I'm not an expert or a professional, but I can give some general answers from what I remember from my undergrad days.

This is a question that has been asked and discussed for a very long time, and is one of the central big questions in ethics. However, there is no definitive answer. This topic remains very, very active to this day, and gets more complicated as time goes on to boot.

Much of philosophy relies on a sort of looking-backwards technique where you first figure out exactly how you intuitively feel about a given topic. From there, we try to figure out why and maybe how we possess that position. An intuition is your gut feeling - it's how you naturally and immediately feel about something without thinking, only reacting. This is where its value largely comes from in philosophical discussion. The intuitive position a rational, normal human being might possess is the baseline. Ethics relies quite a bit on our own intuitive feelings about the various topics under its purview, and often our intuitions about many of them are just very, very difficult to explain.

Generally, the "species" answer, that we prioritize humans over other animals because they're fellow humans, is the first answer people give when asked. Philosophers tend to not like it because, frankly, it's a pretty boring answer that's also not particularly insightful. Simply citing evolutionary success/advantage as a reason and calling it a day seems to not do the topic justice, after all, and it gets murky when you go deeper into modern life/civilization and stuff like star trek aliens (i.e. non-humans that are of similar intelligence).

Intelligence is definitely an important consideration. I can't remember anything on intelligence as intrinsically being valuable enough to justify species discrimination, so I'll skip straight to the practical considerations. Being surrounded by more and more intelligent people, and especially ones you can better communicate and interact with, definitely has its advantages. A community full of more and more intelligent members leads to innovation which leads to improvements in quality of life, etc. The counterargument to this, as you pointed out, is that we often put great value on members of society that aren't productively intelligent, like the developmentally challenged and handicapped and the elderly. In fact, along with children, often people tend to view those people as needing an especially high degree of protection and shelter. This largely rules out intelligence as being a necessary factor to any significant degree, and especially strips the pragmatic aspect of it. On the other hand, you could perhaps argue that even the least intelligent human is more intelligent than the smartest animal, which is possible, but very arguable.

The "sense of self" you describe is often referred to as consciousness. As research into animal behavior and biology advances, it gets harder and harder to use consciousness as a distinction without pushback. Primates, elephants, dolphins and whales all exhibit a lot of surprisingly intelligent behavior that point to the presence of a sense of self beyond the instinctual. Depending on how you choose to define consciousness, you can strengthen and weaken any claims towards animal consciousness, but I don't know if you can completely rule it out. Regardless, why consciousness is important kind of circles back towards intelligence, that a higher level of consciousness points to a higher level of intelligence, which is already probably not necessary, though it may be sufficient. You could, on the other hand, perhaps argue that consciousness allows for awareness of pain and suffering, and a higher level of it leaves a higher capacity for experiencing it. Consciousness still falls short when you look at it this way, though, when regarding the comatose and perhaps even infants and the mentally handicapped.

Inevitably, many people will call upon the potentiality concept. This doesn't really apply to those born mentally retarded or with something like down's syndrome, but for children especially, and even those in comas, this can be a powerful argument for their value. This doesn't really apply directly to species discrimination, though. Potentiality doesn't stand completely on its own, so to get to this point you have to make at least make the concession that human beings, by whatever feature/characteristic, has some kind of value. The basic idea is simple: children and the comatose are valuable because they have the potential to be fully complete persons.

Actually, let me digress a bit here. Personhood is a very important concept in this discussion. A "full" person is someone who is the epitome of the intelligent, conscious, rational and functioning individual. Your typical healthy human is a full person. It's a very loaded word that sort of packages everything that makes an individual as valuable as they can be without actually knowing what goes into the package.

So to go back, children, and certainly infants, are not considered full persons by any measure. They lack compared to a fully developed adult in almost every way. Despite that, they are almost always considered more valuable and deserving of protection and consideration than nearly any adult in nearly any situation. It makes immediate sense to attribute this to their potential.

And...I've run out of time to continue writing this, so I'll just skip to the end.

The big conclusion I was trying to get to is really that there's just no single answer, and there's certainly no single black and white answer. You'll be hard pressed to find anyone who believes an animal's life is ever more important than a human's except in the most extraordinary situations, but people do still attribute a lot of value to animals. Animal rights activists will attribute a lot more value to the life of any given animal than your typical steak lover will, but even the steak lover probably wouldn't say you can kill monkeys and lizards without any consideration. Intelligence and species familiarity definitely play a big role, neither are necessary characteristics to qualify as a full person.

But yeah.