What's to prevent the buyer from demanding their money back if there was no formal agreement that they were buying it, even if they dropped/crashed it on the test ride?

Make the agreement ahead of time. If they don't want to follow it after they drop the bike you have the money and they have the bike. The only thing that's up to them is whether they want you to sign over the title, but the transactions pretty much over.

Also what's to prevent the seller from saying "I've got your money now, that means you bought it." if the potential buyer takes it on the test ride and decides to not buy it(with no drop/crash this time)?

Bring someone with you when handing over thousands of dollars to strangers. Also if you don't sign to have the title transferred it would just be theft if the seller wouldn't give you back your money.

The main thing is to not do transactions with sketchy people and to take a picture of the other person's ID in case smething happens.