I think that non-monogamy brings it's own challenges to people who grew up with toxic parenting, but I also think that non-monogamy has particular problems with people always trying to prove to themselves that they can "do" non-monogamy and always having to be constantly happy as a means of proving that.
Socially, it's accepted that monogamy can be a miserable drain. You're allowed to be unhappy about monogamy and still be seen as it being a valid choice. But non-monogamous people have this happy PR crab bucket mentality where they tell themselves and each other that they have to be almost always happy with non-monogamy or they aren't really monogamous.
I wrote a few things about toxic parenting and non-monogamy that might help you:
9 ways toxic parenting impacted your non-monogamy 9 strategies for counteracting toxic parenting in non-monogamy
For me, I avoid making rules in general, unless they actually help. I feel like a lot of people, regardless of whether they are RBBs or not, make rules in non-monogamy as a way to ease their anxiety and they treat the symptom rather than the disease. For example, some people put the rule of, "You have to check with me before you sleep with someone" as a rule in their relationship. And that seems like common sense, but it could really blow up in practice.
Like my domestic partner does casual things sometimes. I don't want to be phoned up and asked permission at 2AM about whether or not he can sleep with someone. Also, if he asks me for something, especially due to being an RBB, I'm going to be inclined to say yes. I'm not going to want to make him unhappy. I'm going to prioritise his happiness over mine, even when it might cause resentment. I also have learned by being an RBB to ignore my feelings and assume they are invalid or I'm blowing things out of proportion. So I would say yes because I would assume the feelings I have about it aren't valid. So it all would just blow up in my face.
People make that rule because they want to make sure their partner cares about them and checks in with them. I feel like that should happen regardless and instead of making a rule like that, you should work on checking in with each other constantly. Because having that rule in place isn't a guarantee that you'll be checked in on. And you might actually need that more than being asked permission. Because you can easily get someone who follows the rule to a T, but once they have your permission, doesn't check in or care at all.
So that's just one example. In the end, I say, the only rules that matter are these: what you can do and what you can't do. I can allow my domestic partner to make any choices he likes in his life but I can't pretend to be happy when I'm not.
I also accept that I might feel shitty and don't beat myself up for it. Being an RBB, I was not allowed to upset or unhappy. Add that to non-monogamy and I will literally beat myself up for having bad feelings and therefore not being up to the standard of non-monogamy. I feel like the culture of non-monogamy encourages this and a lot of the way they teach you to deal with feelings is SO unhelpful. There's this attitude of if you have bad feelings you need to "deal" with it and that's it. You shouldn't burden your partner with these feelings. And that really is an unhealthy reinforcement of what many RBBs grow up with.
And in fact, to point to the last paragraph you wrote, I have THE HARDEST time trying to admit when I'm unhappy or when I need help with my domestic partner. Autism also causes me to struggle with identifying my own feelings. So sometimes I don't realise I'm unhappy until awhile down the line and then I start to question whether or not I should be unhappy thanks to my RBB upbringing and then there's also some times where, due to the catastrophising my dBPD mother did, I get super paranoid and jump to conclusions.
So in the case I described in my example above, I was super upset because someone I really admired had died. I'd never gone through this before so all of the emotions I felt were super confusing. My domestic partner wanted to go out to a party that night, and I understand that he's more extroverted than I am and so I didn't want to deprive him of going to the party, so I tried to manage my feelings on my own.
But it didn't work. I really needed support. I was scared to ask for help for a lot of reasons. I've had moments in my life where I've tried to get close to my dBPD mother and have been completely rejected. One time I remember I sat next to my mother and smiled at her after my stepfather had gone in for a dental appointment and she looked at me like I was the worst sort of scum and said, "You know when he gets back you're going to move, right?".
I always innately believe that any attempt I make to ask for help will be rejected and I'll end up worse. So instead of asking for help, I'll try and manipulate situations so they end up in my favour. In this case, I tried to ask my domestic partner to have dinner with me before he went out. But actually, travelling all the way to my place and then to the party would have been completely out of his way. I kept sort of asking without asserting anything and he, not knowing I was asking for support, just said he didn't want to.
My brain went into panic/paranoid mode. A little voice inside goes, "See, he doesn't even want to be around you because you're BORING? Why can't you just go to the party with him? Oh yeah, that's right, it's because you have sensory processing problems. You're the only one with those problems. You're a freak. No one likes you." and then there's this other little voice that goes, "Can you believe him? He can't even spend an hour out of his way to have dinner with you. He's just like your parents. He doesn't actually care about you, he just wants you around when he wants you." I got actually quite angry with my partner. I was crying on the phone and I was like, "I don't ask for much. I just want dinner with you. I've had a rough day and--" Literally my partner interrupted me and said, "Do you need me?"
My inner instinct was to say, "NO I DON'T NEED ANYONE" but I caved and said, "Yes." And he came right over. We actually ended up going to the party later together and had a good time. But this was so hard for me. I had to almost be forced into accepting help. And over time I'm learning that it's okay to ask for help, but I still struggle with it to this day. I don't think it's something you can get over very quickly. It's going to take time.
The best thing I learned about anxiety was to accept that I had it. As silly as that sounds, it opened up a whole new world for me. Before, I would see every panic attack as a failure. I would berate myself for feeling anxious and it just made everything worse and made me feeling like Sisyphus, pushing the bolder up the hill constantly. When I reframed my thinking and said, look mental health is a lot like physical health. I don't berate myself when I feel tired (I have a thyroid condition), so why berate myself for being anxious. I can't control that I have anxiety. It will come and it doesn't mean I've failed. I try to take that step when it comes to my emotions. I try not to beat myself up for feeling scared or worried and just accept that I feel that way and focus on trying to cope with it. That really helps.
I wrote a few other things on non-monogamy that might help too:
What anxiety taught me about non-monogamy Non-monogamy and fear A relationship is not a skill
Hope it helps!