I'd like to say you're doing well to ignore it but it seems as though it is still bothering you.

Therefore, I take it that you are not ignoring it but rather you are suppressing the outward display of discomfort. Inside, you are letting it get to you. If you are comfortable with who you are/what your values are, why should any outward influence bother you?

There are certainly arguments that could be thrown against his claim but that would invade his personal values. It wouldn't be so wise to stoop to his level. Rather, just think to yourself: "I know what I know and I don't need to prove anything".

With such self-assurance, you should be able to ignore your brother's attempt to wrong you, yet have the ability to react appropriately and ideally, maintain positive relations with him.

Your brother may never understand an alternate perspective that opposes his views. That's okay. But hopefully one day he will grow out of his naivety just enough to understand the importance of respecting another's point-of-view.