You're hearing it correctly.

The answer is that humans are lazy creatures. Speech in most languages has a tendency to blur and slur words in order to more easily pronounce them. The less effort you need expend in actually moving your mouth muscles the better, and this is especially true as you tend towards lower formality.

As you have observed, sometimes this happens and sometimes it doesn't, because the proceeding sounds don't always lend themselves to being blurred (either because blurring would lose meaning, or because it doesn't actually make the speech any easier) I don't have the linguistic knowledge to explain when exactly this is or isn't, but I'm sure you could research the topic if you wanted.

In my opinion, it's absolutely NOT something you should spend too much time worrying about. If you pronounce あの人 as a-no-hi-to with each sound clearly enunciated, that is perfectly correct and you will be well understood. Knowing when and how you can relax your pronunciation is something that should come naturally through practice in speaking and listening.