I was thinking about the individuals responsibilities (and rights and their limits) in terms of other individuals- As opposed to the individuals responsibilities to the state. I guess it's a sort of blurry line- withe argument that the state represents society as a whole.

Well, yeah. But the Bill of Rights is also part of the legal system. We're not the real audience of the Bill of Rights, the government is. As you said in the OP, it's the other side of the coin of responsibilities. Unlike our responsibilities laid out in the legal code, it's the government's responsibilities to us that are laid out in the Bill of Rights.

But often (ideally?) when we talk about 'rights' its about individuals, with no state intervention (until stuff has already gone wrong). This is the context that limits and responsibilities matter.

Well, that's more a fault of misunderstanding what rights are. Or, perhaps, confusing moral rights (how we should voluntarily choose to treat people) with legal rights (how we should choose to treat people, or else). With regard to the law and legal rights, the Bill of Rights only protects us from the state.

Some people wander around whining about their rights in ways that just prove they have no idea what their rights are. The last couple of months have seen a series of huge shitfests over reddit not being some kind of bastion of free speeches!!!! But first amendment free speech doesn't, and has never, protected anyone from anything except action by the government of the United States. It's the right to speak my mind without censure by the government, not an obligation for everyone else to listen to me.