So how do you fix it? Do you tell some students no and let others go?

You aren't dealing with adults in this situation, you're dealing with kids who will jump at any chance to point out the fact that you're singling them out then screaming bloody murder because of it.

In the adult world, no, individuals are expected to take responsibilities for their own actions as adults. If you walk out of a class in university then thats on you, nobody cares, you're missing the lecture and only hurting yourself. We aren't talking about airports or blanket bans for entire populations here, we're talking about a school, a place where the only adult in the room is responsible for educating and keeping track of all of the minors in the room. One slight slip and they're liable to be fired when parents form a lynch mob.

Do you let kids clearly take advantage of you time and time again so they can wander the grounds doing what they like, or do you tell them they can't leave the room because they had a chance to use a bathroom before class? Because both options leave you wide open to landing yourself in trouble as a teacher.