While physical capabilities (or indeed, mental capabilities) certainly could have been one factor, I don't imagine it was ever the only or primary factor determining who got what job. The sort of system you describe is a very modernist, rational system that might be more appropriate in describing a planned, meritocratic government system rather than the more organic development of early agricultural societies.
Instead, resource availability, restricted knowledge, and primacy of place were probably some of the most important factors in deciding who did what. For instance, there are several good examples in the U.S. Southwest were migrants moving into an area are forced to live on less-optimal farming land. This is because the first people living in a location of course settled on the best farm land and claimed it as their own. Consequently, the people living on marginal agricultural land would be encouraged to supplement their "income" by specializing in producing certain goods, such as pottery. This may have been the case in southern Arizona with Hohokam specialization, at least in the Phoenix basin, although this is still very much debated.
On the other hand, a different kind of model would say that the primacy of place is most important. The first farmers in a location have this primacy of place - that is, their claim to the best farm land due to their ability to trace their ancestry to other people who had lived on that land - may be an important component in allowing them to occupy prestigious social positions. Their access to greater resources, due to their location on prime farming land, may have helped increase their wealth relative to their neighbors, further putting them in positions of power in society. These people may want to invest in markers of their status as the original inhabitants of their location to secure their ownership of the best land. There are a huge number of cases in neolithic (i.e. early agricultural) societies across the planet where ancestor worship seems to be an important component of religion. If the land you own is tied to who you ancestors were, enshrining them becomes an effective way to increase your social power by cementing your hold on productive lands and thereby increasing your wealth. For example barrows and other types of burial mounds in Neolithic Europe.
This association with early religious practices may also then have translated into these individuals becoming a nascent priestly class due to their command of ritual and special knowledge about the supernatural.
This is a very brief summary of a very complex topic and I'm certainly glossing over a lot of the variability, but, to sum it up, it was social factors, not physical or mental ability, that probably most influenced who ended up in particular social roles. Factors like resource scarcity and productivity along with social rules about property ownership and the development of early religious beliefs about ancestors may have been the key components in delineating social roles in early agricultural societies.
Perhaps other users could elaborate with more examples.
Sources: Harry, Karen G.
2005 Ceramic Specialization and Agricultural Marginality: Do Ethnographic Models Explain the Development of Specialized Pottery Production in the Prehistoric American Southwest? American Antiquity 70(2):295-319.
2002 James Whitley. Too many ancestors. Antiquity 76:119-126.
2001 Kuijt, I., Place, Death, and the Transmission of Social Memory in Early Agricultural Communities of the Near Eastern Pre-Pottery Neolithic. Archeological Papers of the American Anthropological Association, 10: 80–99.