This is a pretty open and close case for libertarians.
The only rights you have are Negative Rights and they are not granted to you by anyone or any government. The government is in place to protect your Negative rights, or Rights that require inaction by others.
All people are 100% equal in all aspects of life. No group of people can justly claim more rights over other groups.
If an individual can voluntarily associate for any reason, then a business can do the exact same.