It's like any other job, but somethings you keep a certain cognitive dissonance with. I work in Canberra and basically half the population has a clearance on some level. So I guess I am biassed as I don't really get exposed to many people who doesn't immediately know to not push you on details.
That being said, I don't exactly have a high clearance or see anything that actually matters. You just spend a bit of time figuring out what can be spoken about or not. Then stick to it. And honestly. I am yet to date anyone who would care anyway. Most classified material from an IT level is actually boring as hell to the majority of people; and they just don't care to hear it.
So I guess the TLDR is: It's a non-issue for me at least.