Derrick Jensen's 'Deep Green Resistance' movement have an interesting response to this, although lots of people regard it as transphobic, and DGR also call themselves Radical Feminists. They posted a response to this on their website which is worth a read. Here's some key points from it:

Patriarchy is a caste system which takes humans who are born biologically male or female and turns them into the social classes called men and women. Male people are made into men by socialization into masculinity, which is defined by a psychology based on emotional numbness and a dichotomy of self and other.

We are not “transphobic.” We do, however, have a disagreement about what gender is. Genderists think that gender is natural, a product of biology. Radical feminists think gender is social, a product of male supremacy. Genderists think gender is an identity, an internal set of feelings people might have. Radical feminists think gender is a caste system, a set of material conditions into which one is born. Genderists think gender is a binary. Radical feminists think gender is a hierarchy, with men on top. Some genderists claim that gender is “fluid.” Radical feminists point out that there is nothing fluid about having your husband sell your kidney. So, yes, we have some big disagreements.

Radical feminists also believe that women have the right to define their boundaries and decide who is allowed in their space. We believe all oppressed groups have that right. We have been called transphobic because the women of DGR do not want men—people born male and socialized into masculinity—in women-only spaces. DGR stands with women in that decision.

We are not gender reformists–we are gender abolitionists. Without the socially constructed gender roles that form the basis of patriarchy, all people would be free to dress, behave, and love others in whatever way they wished, no matter what kind of body they had.

Link if you're interested: http://deepgreenresistance.org/en/who-we-are/faqs/radical-feminism-faqs