Regarding rape culture, no one seriously asserts that it is to be held literally responsible for all rapes that are committed - the notion of rape culture doesn't disclaim any responsibility on the part of the rapist him-/herself. Rape culture does, however, play two key roles as regards your argument:

• Promotes a culture of victim blaming. This works against all genders (the dynamics between them vary, obviously). Rape is a dehumanizing act; holding a victim accountable for being raped, whether by a stranger or an acquaintance, isn't comparable to blaming someone for leaving their doors unlocked as it is to blaming a Vietnamese villager for accidentally wandering into an abandoned minefield. Yes, you could invoke decisions that the victim in each case made that contributed to their fate, but ultimately, no matter how blind drunk someone gets at a party, it's entirely the rapist's responsibility to deny them their humanity by violating their body. This principle applies equally to deliberate acts of violence and to unwitting failures to secure consent.

• Blurs the definition of rape and implies that there are acceptable scenarios in which rapes can be somehow justified. You ever see Sixteen Candles? Part of the plot of that movie involves the blockhead who wants to get with Molly Ringwald literally handing his passed-out drunk girlfriend to Anthony Michael Hall with the exhortation to "have some fun" with her. He is clearly conscious and lucid; she is not. They wake up the next morning having had sex under these circumstances, and it's played up for romance. What the fuck is that?!

That's only one famous example but there are countless ones. No one is commanding anyone else to rape or saying explicitly that it's an okay thing to do - but there is a definite cultural climate (in the US, if we must be pedantic here) that tacitly condones certain forms of it under certain circumstances.