This is an interesting question, and I would be interested in the answer but I strongly feel that it needs to be focused differently.

the Nazis wore skull emblems on their caps

Sure, but skulls are ridiculously common for military badges and emblems, the US military uses skulls, death, destruction, mass death, and fear as standard-issue for their regalia, as do many other countries armed forces, so it seems to me that picking out an instance where certain military groups which are choosing to use fear and intimidation in their 'branding' does not set them aside from other military groups; as 'fear, death, and destruction' are pretty much stock-in-trade.

ISIS puts heads on pikes

Saudi Arabia chops people's heads off nearly every day, and uses gruesome forms of display of their bodies, and Iran practices mass hangings; this type of fear tactic is common in the region.

shouldn't they make themselves more appealing to everyday Germans/Sunni Arabs

You may be cherry-picking with this, the Nazis put an enormous amount into making their ideology appealing, although their dark and vicious work has lasted more through history they also made more typical appeals to popularity.

This comment is probably just going to get deleted by the subreddit mods, but I have a collection of images posted to Twitter by ISIS that show their humanitarian efforts (feeding women and children, opening schools, building water purification plants, etc), that I would find if I thought anyone would see it (other the mods and hopefully OP) but my point is that both ISIS, and the Nazis, did make a concerted effort to appeal to the good side of people; the side that you have chosen to see is the dark side, and that is common to violent/militant groups worldwide, not just those more commonly associated with villainy.

Anyway, I think this question needs to be re-framed to:

How often is the use of a villainous, threatening, or dangerous-seeming imagery a positive propaganda/media tactic, and how often, and why, does it backfire?