it's ludicrous to put the onus of even figuring out when to pay sales tax let alone actually paying it on the buyer.

This is due to the Commerce Clause of the United States Constitution which says the United States Congress has the power to "regulate Commerce with foreign Nations, and among the several States, and with the Indian Tribes."

So if the buyer and seller are both in, say, Ohio, the transaction falls under intrastate commerce and is subjected to Ohio sales tax because Ohio regulates commerce in Ohio.

If someone from Ohio buys something from someone in California now the transaction is interstate commerce and falls under the purview of Congress, not Ohio and not California. So Ohio can't charge its sales tax unless the seller has a physical presence in Ohio. Quill Corp v. North Dakota (1992) is the case law for this scenario - the "physical presence."

So states say "Oh! I can't regulate interstate commerce but I can levy a use tax** on the people who live on my state!" Historically, enforcement was nil on all but large items that needed to be registered (cars, boats...). After people started buying stuff online many States started to put line items on their tax forms for the amount of the items you bought from out of state vendors in order to collect the tax in that way.

**a use tax is a tax on the "use" or "consumption" of items. Basically like a sales tax but the responsibility of pay it is on the "user" not the seller.

Having Congress regulate interstate commerce is actually really, really important. Historically, one of the things that lead up to the Constitutional Convention was states putting large taxes on goods traveling outside of their state. Having Congress regulate interstate commerce lead to harmony between the states.