The "patriarchy" isn't some secret cabal of people. It's a way of structuring a society, and it goes back a long time in history.

Patriarchy is a social structure where men are "in charge." It goes down to the family level, where the father (or "patriarch") of the family is considered in charge of the family, and the wife (or wives), children, and property belong to him, and this ownership was transferred through patrilineal descent. In most of the world, for most of history, that ownership was literal - a man very literally owned his wife and children as if they were property. Although this legal/literal ownership is slowly going away in some parts of the world, it's not easy to just undo thousands of years of social organization around these concepts.

This idea of "father in charge" then radiated upward through all power structures. If a woman couldn't be in charge of her family, she certainly couldn't be in charge of society. Wealth and power were transferred from father to son because of patrilineal descent.

The idea of fatherhood and patrilineal descent is the root of patriarchy. For patrilineal descent, you have to know which children belong to you. But in a world before DNA testing, how do you know which children belong to you? The only way is to totally control a woman. So a woman was controlled by her father, who owned her, until she could be passed off to another man as a wife or concubine, who would then own her. But at the end of the day, it's all about controlling reproduction.

And then, like any method of social organization, ideologies develop to justify and enforce it. When people talk about patriarchy, they're talking about both this method of social organization, and the ideologies that justify and enforce it. For example, most major world religions explicitly hold that women are inferior and/or subject to men in some way. Legal systems were created to limit women's freedom and reinforce their role as property. Cultural norms developed to reinforce these roles for women and, over time, just become "the way things are."

And just to give you an idea of how recent some of these changes are: Marital rape did not become illegal in all 50 US states until 1993. Less than 30 years ago, a woman did not have the right to say no to sex with her husband. In Japan, a woman seeking an abortion still needs the consent of the presumed father. The whole abortion "debate" really comes down to whether or not women get to control their own reproduction, or if that capacity belongs to men. Either specific men, or the men who still largely control social/political/legal power structures.