It makes mire sense if you think of every state as a nation state; each with their own constitution, laws, and judicial systems (which is true). As it's a State's Right, and not a Union (US government) Right, there isn't a "Federal Police". Because of that, you have federal agents for separate, specific purposes.
 
The FBI, for example, is part of the IRS. Their scope has expanded since Hoover's time, but the concept is the same. ATF deal with interstate transfer of illicit substances, and therefore fall under the interstate commerce clause that grants the Union power of oversight.
 
City police and Highway Patrol deal with the obvious; whereas Sheriffs come from Britain for the one that manages the shire. Sheriffs manage unincorporated land policing and other similar issues for the state.