His driving was certainly suspicious and definitely casts a grimmer light on the contents of his trunk, but I keep rope, a burlap sack with some random fishing gear thrown in it, and even a hatchet in my car pretty much all the time. I purposely put the hatchet which has a bit of a hammer type poll on it into my car because my state has Good Samaritan laws in regards to smashing windows to save pets stuck in hot cars. So while a mallet is strange, it's possible there's a good, non-murderous reason for it.
Especially if you live in a more rural area, I wouldn't be too surprised if a lot of people didn't have a lot of similar things in their car be it for camping or emergency reasons.

While my hope is that he was just a confused driver or something, it definitely does seems like he was up to something fishy. Either way, good for you for trusting your instincts and remaining assertive in the situation.