Your property rights and much of the infrastructure you use are sustained by society at large. That gives society some say in how businesses operate. We don't have a problem with safety and health regulations, for instance.
Society =/= Government
“Socialism, like the ancient ideas from which it springs, confuses the distinction between government and society. As a result of this, every time we object to a thing being done by government, the socialists conclude that we object to its being done at all. We disapprove of state education. Then the socialists say that we are opposed to any education. We object to a state religion. Then the socialists say that we want no religion at all. We object to a state-enforced equality. Then they say that we are against equality. And so on, and so on. It is as if the socialists were to accuse us of not wanting persons to eat because we do not want the state to raise grain.”
Infrastructure that is made by government by taking my property (in taxes) without my consent which somehow then justifies government to dictate how I use my property. Would no roads be created if not by Government Decree? I doubt it. It could priced into home, business could lease naming rights and advertising on roads to maintain them (bill was just introduced in Missouri to do this), or you could have toll roles. Are any of those perfect? No, but is government management of our roads and bridges that great now? No. I do not think this somehow justifies them infringing on my property rights and my right to freely interact and engage in exchange with others.
We don't have a problem with safety and health regulations, for instance.
O there are plenty of people with big problems on this. There are countless examples I can give you of Government enacting those types of regulations to stomp out competition, in fact the creation of the FDA was lobbied for by big butchers to help stomp out competition. People say they want more locally grown food but in the same sentence want more regulations put on farmers which leads to the mega corporation farms they decry. For example if I'm a farmer and don't take my meat to an FDA approved butcher (which of course costs more) I can't legally sell you my meat. SWAT teams have been sent to farmers houses for selling raw milk. Is that how a free society operates? If I want to sell you milk and you agree to buy it and I have not misled or defrauded you in anyway why should it be illegal for me to sell you that milk? Is that a free society? Take a look at new FDA regulations for E-cigs that also do nothing to protect the consumer and favor special interest. The only morally justifiable system is one where individuals are free to engage in exchange without coercion, fraud, and government intervention.
And for Safety OSHA has ridden the wave of safety improvements that was always trending down faster and have now leveled off. Capital investment and new technology have improved Safety not government agencies. Companies aren't evil, they don't want to intentionally poison everyone and they don't want their workers to die. You see this with cars as well the Government only mandates features after they've already been introduced years before but instead of waiting for the tech to get cheaper they then mandate it causing an increase in the cars price leaving older cars on the road longer which are less safe then new cars without the mandated features.
Restricting someone from something because of their race or gender doesn't harm them?
How does it harm their life liberty or property? Does it hurt their feelings? Sure but you don't have a right to not get you're feelings hurt. Now should government discriminate? No, should private individuals be free to as well as businesses, yes. We just had a Judge say a bar owner can't discriminate against pregnant women and refuse to serve them alcohol, yes the bar owner would be fined if he refused to serve them.