Honestly, for me, it comes down to who sees me eating the leftovers.

I don't think it's at all hypocritical for vegans to eat non-vegan leftovers that will otherwise be thrown away. It doesn't increase demand to kill animals by eating leftovers.

That said, not everyone is willing to hear your ethical argument. Many people will savor ^{no pun intended} the opportunity to catch a vegan eating (leftover) meat and point out apparent hypocrisy, apparent necessity to eat meat, blah blah blah, etc. And those people won't be willing to hear you out if you try to tell them about moral nuances.

So I will never eat non-vegan leftovers in public, simply to avoid those aforementioned stupid situations with bad intentioned people. God forbid if a famous vegan were ever caught in such a situation.

But, again, I see nothing unethical about it. And if you're around people who understand the nuance of the situation, who understand that you're against the process that created that food and you'd never pay to support it, then go for it.