It's a perfectly valid reason. We like to preserve what we know, and what we have. If we absolutely knew that Solas' plan would work, his world would be back as it was afterwards with all the elves restored then we're effectively trading lives, which is bad - but not particularly bad excepting for that sense of self-preservation. The world would go on, thrive at least as well as it is.

Except we don't even know that the plan will go off without a hitch. That's the other reason people instinctively conserve what they know. Because change doesn't always lead to something better. It could always get worse, even a lot worse. No one, not modern people and not ancient elves or anything at all, could be left standing.

That kind of risk isn't worth it even if you DO like the idea of a world filled with magic and think that Solas' world would be the better one.