Thats interesting.
In Buddhism, Yama (Sanskrit: यम) is a dharmapala, a wrathful god or the Enlightened Protector of Buddhism that is considered worldly,[14] said to judge the dead and preside over the Narakas ("Hell" or "Purgatory") and the cycle of rebirth).
This suggests Yama has some power to rule, make judgement with effect, as a judge in law can send you to prison. Yama can send you to hell. So in the presence of Yama, someone with bad karma, meaning someone who has acted in ways they know they should not, the fear of punishment would arise. That would also align with the frightening image of Yama.
How exactly this relates to impermanence, is also a bit interesting. Because to someone who is clinging or is attached to something, the force that removes or destroys what someone is clinging to, is precieved to be ill-intended or evil. In the way a child cries when someone takes away their favorite toy, they would also learn that that someone, or in this case we use the term force, that force actually has the power to take away. One would then live fearful of having what they are clinging to taken away.
To take it a bit further, one's beliefs or mental imagery, could it even be suggested these are the attachments that one is fearful of having taken away. Just to make it a bit more relatable. The belief system, being the known, although very limited and small - is what is being clinged to, Yama, also representing impermanence, the force that takes away everything, revealing the unknown which would be very aversive to someone clinging to the known.
Now here is where it gets interesting, get ready for this plot twist. Yama is actually the salvation, the force of impermanence is the cleansing. It's all back-wards, Yama is the good guy. The force of impermanence is a blessing, but it looks like a curse when one is attached.
It is like the garbage collector coming to collect all the trash, but the people believe their trash is actually treasures, so they freak out when he takes the trash by force. So the people started calling the garbage man evil and got scared of him. Bless the garbage collector.