Yeah, I went through the same thought process, after a long time I decided I would join the stormcloaks not because of Talos or the Nords but because the empire is cowardly and weak. Rather than fight the elves they fight their own people and bleed themselves white. But the emperor knows that if he fights the elves he could lose his throne, if he fights Ulfric it's only the soldiers who have to die. In the long run he is weakening the race of men so that he can hold onto his domains in Skyrim a little longer. It is the hallmark of a decrepit and dying institution that sells its future to keep up the present status quo. If the empire won't fight the dominion, it needs to get out of the way and let the Nords battle it out with the Thalmor themselves. If Hammerfell can do it, so can Skyrim.