You just learn to ignore it for the most part.

The US is less than 2% Jewish, most people who talk about Jews have little or no experience being around Jews.

People have been suspicious of Jews forever. It's partially due to the historic lack of assimilation by Jewish people, especially the more Orthodox groups, and that at least in my experience, there is a minority of Jews that do have an us vs them mentality among the more zealous groups. This can be very off-putting and difficult to understand for anyone, especially if you don't know much about our religion or history.

People fear things they don't know about or understand. Just like other how religions get a bad name because of their fringe elements (think Wahhabis or the Westboro Baptist Church), people tend to judge Judaism based on stereotypes of our more Orthodox elements, or tired old stereotypes of that have been perpetuated for decades.

The conspiracy theories definitely don't help.

Anti-Semitism has been a very pervasive part of history, and trying to pin down why casual Anti-Semitism is still so acceptable is difficult.