I've realized that I don't think I'm ok with the open relationship.

You never were ok with it. You pushed yourself to be ok with it due to your lack of sex drive. I don't think either of you want this given his initial hesitation. In order for polygamy to work you need to be polygamous before you lose your sex drive. Otherwise it's just you convincing yourself this is what you want in order to keep your husband. I suggest stopping the polygamy immediately, because it is only going to cause you more hurt. You need to address the sex drive issue or you need to start thinking of separating. Don't use polygamy to solve your sex issues, it's only going to blow up in your face.