Well, I'm not the one here who cares so much about upvotes or downvotes. So from my perspective, to answer your specific question, since you asked me directly:
No. It wouldn't "be something" in my own life!
It would be very insignificant... to me at least. But sure, fair enough: to you: I guess it would mean a lot if that happened, from the sound of it, again because you really (REALLY!) seem to be focuses upon and consistently harping on "votes".
Thus, if I do end up with more downvotes than you, then...
Great! Hooray! You can celebrate!
And I can... shrug!
(Although if you celebrate by throwing a party with free Champaign, then I'll totally join in with the party and eagerly help you celebrate my loss!)