how much of their popularity is due to tacticool and military association and how much is the gun itself?

Yeah, I hear ya. As someone who owns no small amount of surplus gear/uniform bits, I definitely think being associated with the military has some effect on their popularity. But I also think it wouldn't be a particularly different world - the move to less powerful ammo in a military context was coming either way. Bullets can't be designed to be good at killing people but not animals and vice versa, so whatever round some fantasy other rifle used would almost certainly be similar to varmint rounds even if they weren't in the lineage like with 5.56. Would the fantasy military rifle be more popular than an AR? Probably, but it also would likely be practical in all the same ways an AR currently is. We can look at things like the AR-18 and various derivatives of it down to the Sig MCX. That rifle was not adopted by any military, yet remained reasonably popular for about 60 years until now, when the army has only just adopted what is effectively a clone of it. Hell, there are now AR-18 style uppers you can throw on an AR-15 lower, because the AR-18 was popular but not as cheap as an AR-15. I expect we'd see something similar in the case of an AR-15 in this mirror world where it wasn't adopted by the military.