every single time a university allows or invites anyone on their campus they are making this judgement.
For one thing, those decisions are rarely made at the university level. The overwhelming majority of speakers invited to campuses are invited by departments, centers, student groups, etc. There's no one person or one group of people making these decisions for everyone.
You're missing the point, anyway. It's one thing to not invite someone to campus because you don't find their speech valuable. It's another thing entirely to take it upon yourself to decide that the speaker that someone else invited to speak that their event shouldn't be allowed to speak.
do you genuinely think there is some unmined seam of valuable ideas that is being prevented from speaking up in academia because it's too politicised?
Yes, and liberal academics like Jonathan Haidt think the same. I also know that I, as a moderate, have views certain issues that I will not publish on or make widely known at least until I have tenure, and I've heard others express the same.
yes, and the 'debate' i referred to earlier is for the most part around exactly what fits into the first and second categories.
Yes, there should be debate about that, but in order for their to be a healthy debate, everyone involved needs to be especially patient, charitable, and yes, willing to be offended. A big problem in debates about issues like race is that so many people on all sides want to shut down as soon as they hear something that offends them, but there's simply no way to have the conversation without facing the risk of discomfort.
even the art department would draw lines i am sure.
Given the inciting offense and outrage is often considered to be an explicit goal of art, art departments and art institutions tend to be very lenient with the lines they draw, and a lot of what comes out of art departments is remarkably offensive to a lot of people. To Yale's credit, though, they did prevent Aliza Shvarts from using her self-induced abortions as her senior project unless she stated publicly that she faked them, although even as fakes, that sort of thing is shockingly offensive--and intended to be--towards anyone with pro-life views. Even still, being pro-life, I would defend the freedom of a student or faculty member to do a project like that (assuming, that is, that it is faked) however repulsive, borderline deranged, and basically worthless I find the project to be.
i genuinely can't envisage this scary environment you are talking about
And many white people can't imagine the scare environment that black students are complaining about. In most cases, that's probably a failure of the imagination of white students, and by the same token, the failure of liberals to be able to imagine how scary it might be for conservatives in an overwhelmingly liberal environment is probably more often than not a failure of liberal imagination. It's hard work for those in the majority position to imagine things from the perspective of the minority. Your concern for racism ought to have taught you that.
This isn't just about "uncomfortableness." The academic job situation is extremely precarious these days, and getting a tenure-track job and advancing to tenure relies heavily on personal connections and having a research a research agenda that appeals to the right people. If you make yourself look like an outsider by taking up morally unpopular positions, you can put your career chances in jeopardy.
When you're a grad student or a junior faculty member (or heaven forbid, an adjunct) and you hear your department chair casually joke about how brain dead people who hold position X is, and everyone in the room laughs in agreement, you most likely aren't going to speak up if you hold position X. And that sort of thing isn't particularly uncommon, and one shouldn't expect it to be particularly uncommon in any setting where one perspective is overwhelmingly dominant.