In my world, anyone COULD, theoretically, have magic. Magic isn't some external concept that you can study in a tower (or you could, no one does). Magic functions as an extension of your identity. To do any BIG magic, your identity needs to be subsumed into some larger whole.

An example: there's a tribe of people who spend their lives trading up and down a network of rivers. Women regularly go on shore to barter and trade with the land-dwellers. men live their lives on the water. They swim, they fish, they cook, they fix the boats, they read the river. They rarely, if ever, go on land. Their lives are entirely defined by their boats, their river, and their community of (at most) 100 people. A woman in this community might be so persuasive in haggling that it seems almost superhuman. But say she caught a deadly plague in town, and spread it to her community. No one dies, and in fact everyone miraculously gets better after two days. Because the healer (who is always male) refuses to let HIS people die, and does not acknowledge the existence of this foreign "plague" bullshit. On a yearly basis, 10 men from different communities gather together and WILL the entire estuary system into flowing backwards for a month, so that the entire tribe can travel back up the rivers with ease. to have that kind of power, those ten men go their entire life without stepping onto solid land. If one ever did, he would probably faint from the nausea. They would feel like they were being separated from themselves. The healer who can WILL his community into surviving a plague has spoken maybe 10 words to someone not of the river people. And he doesn't want to.

To the west, there's a mountain range full of gold ore. No one's mining it, because the people who live there don't want gold mines. A nearby kingdom tried to invade once. Their campaign into the mountains lasted a month. In that month, more landslides occurred than in the past 3 years combined. And every single one of them hit the army camp.

Magic doesn't HAVE to be tied to a location, but the more specific and concrete your affinity, the stronger the magic. The strongest mages are the people who are LEAST interested in conquest. To desire something is to lack it, and how can you BE what you've never had?