Nitpicking the terminology is fine, because semantics does not have the certainty of mathematics, but it's hard not to suspect those who do of only to serving their prejudice against Jews.

Anti-Semitic is the historical label for Judophobes. Jews typically see themselves as both a separate religion and a race, and DNA studies actually bear out that Ashkenazi Jews have Semitic origins, despite intermarriage with and conversion of other peoples historically. The prejudice against Jews has been expressed in both religious and racial terms in the past. The Nazis certainly used both - they persecuted people who lived as Christians their entire life just for having Jewish grandparents.

The term "Anti-Semitism" originally comes from Jew-haters themselves. In the late 19th and early 20th century, religious intolerance was incompatible with the reigning ideas of the Enlightenment, but racial intolerance was seen as scientific and endorsed by the colonial powers of the day. It was a re-branding of the same hatred that Europeans had for centuries before, not some pro-Jewish plot.

So most people most of the time, Jew and Gentile alike, have seen Jews as having both their own religion and their own race, even if one was more emphasized than the other at different times. The standard American view of race today categorizes Jews as White or at least Caucasian, but we label haters of Jews as "Anti-Semitic" nevertheless because of historical momentum. I prefer the term "Judophobia" myself because it is more exact, but whatever the words you use the hatred of people simply for belonging to a certain racial or religious group is unjustified.