Before people say it's totally harmless, I had to work really hard to convince my friends that it was a bad idea to drive to the ski hill and go sledding while hallucinating. It was difficult negotiating, since I was also high at the time.
It can lead people to make some really, really unwise choices. Which, of course, is also the same for alcohol.
I don't think it should be criminalized, but it should be acknowledged that its use effectively turns adults into irrational, distracted children, and that it is a burden and potential danger to society if they're wandering around in that state.
If people want to take mushrooms, they need to plan ahead, have activities ready, and IDEALLY, have a chaperone. It all boils down to personal responsibility.