An appeal to authority is fallacious only insofar as it is irrelevant. As others have pointed out, this can be seen in things like appealing to some authority that is not an expert in the field that such an appeal is supposed to support in a given claim. Clearly, for example, what Jenny McCarthy has to say about vaccinations is irrelevant.

It seems, from rummaging around the internet at least, that many see any appeal to any authority to be fallacious, and this certainly isn't the case. I don't know why it is so prevalent or why so many people seem to believe this way. Maybe it is a category mistake. Maybe it is anti-intellectualism or anti-expertise. Maybe bias. Maybe it's a naïve skepticism. I don't know. Perhaps it is an overextension of the fact that just because someone is an expert in a field, it does not mean that what they say is, with a 100% certainty, a 100% the Truth. If we believe in progress of knowledge, same goes for a group of experts and disciplines. It is entirely possible, for example, that climate change is not real or at least what the majority of scientists-- that are experts in the fields relevant to it-- make it out to be. I think it is a slim possibility, but a possibility non the less. It would seem silly to bet on such slim odds. Perhaps people want desperately to be 100% certain of the "facts," but no appeal to authority can genuinely do this.

People can't possibly be experts at everything, so we defer to experts. Its not necessarily a bad thing. I do suppose that a line of thinking about appealing to expertise can be bad if taken to far to one extreme or the other.

So, yeah, your basically right, I think. I do, however, think you're misguided in your framing issue and understanding of the background issues and the possible solution to the problem, but that is not really the heart of your post and falls out of its immediate scope, so I won't address these concerns unless prompted by you.