The basic idea of ethical egoism (EE) is that what's good according to the individual subject's interest, is good, full stop. The ethical egoist would regard someone claiming that "you ought to do something that doesn't reflect your personal interests" as very nearly a contradiction in terms; the claimaint doesn't understand what's meant by "ought". You have to introduce something like "higher interests" to salvage that claim, and then you have to explain how those higher interests are determined if not by individual subjects evlauating them according to their own ethical interests.

As with most ethical theories, objections to ethical egoism come in the form of intuitive counterexamples that such a theory has difficulty accounting for or which lead to absurd conclusions. For example, most of us regard self-sacrifice as ethical, but the ethical egoist regards it as at best nonsensical. It would seem absurd to claim that to save a drowning person is nonsensical if it requires getting your shirt sleeve wet, but the ethical egoist is perfectly well able to perform that action if they wish to and remain consistent; they just don't believe that the action it's ethically required.

EE is unpopular because it doesn't generate many of the ethical obligations that most people take as being intuitively obvious. But EE isn't a descriptive ethics, it's a normative ethics, and its main critique is that most of the ethical intuitions we have are baseless and lacking in justification beyond "I feel that it's true". The challenge ethical egoism raises to utilitarianism, Kantianism and the like, is to justify the basis of their claims according to our motivations; EE is well able to do this on an evolutionary basis, whereas the other alternatives are universally lacking in this respect--accounting for altruism in evolutionary terms puzzled Darwin as well as Alfred Russell Wallace, and it continues to puzzle evolutionary biologists to this day.