I prefer to roleplay with the other people sitting at the table rather than compare dice scores. I understand it can be difficult if you want to play a charismatic swindler, but you as a person are not charismatic enough to convince another player to hand over their spell book; but thems the breaks. You have a world full of NPCs to swindle if you're not a good roleplayer.

If I were in your position I would have not chosen to make a "meta-game choice" and hold onto my book. I'd give it up if my character trusted the guy, or held onto it he didn't. There is no winning in D&D and avoiding conflict does not advance you closer to victory. On the flip side, stealing your spellbook doesn't award him any points, there would be no reason to follow through with the theft if it would wind up being unfun for you.

An important lesson to learn is you cannot remove meta-game knowledge, and the dice are a shitty method of protecting you from it. The best way to protect yourself from it is to learn to not utilize it.

For example, lets assume you know a Kobold has 12 AC because in previous games rolls of 11 missed, and rolls of 12 and higher hit. But in this game your character, with the Lucky Feat, encounters a kobold for the first time. You roll for an attack, see an 11, and decide to reroll. You just meta-gamed and the dice didn't save you.

As for your question about how my table does it. We never roll for interaction between PCs. We always play it out because it's way more fun. And yes we frequently do things as a player we wouldn't, but our character would. And it's often hilarious and always better for the story.