OP isn't talking about a postage charge for the item being sent to them - "they're refusing to refund me the £8.50 postage I had to pay to get it back to them.". But I still think you're right.

Since I think best guess some time in the late 90s or early 00s, when you receive something by mail order (even if you strictly didn't order it) and want to return it, I haven't double checked but I'm certain the company is supposed to collect it from you at their expense. The reason for a change was specifically because of abuses of the law prior to then where it was costing people a lot of money to return wrong/damaged goods that were e.g. sent out as a delaying tactic until new stock arrived.

Anyway, obviously that's not absolute, and it's often impractical for both parties. But returns are supposed to be at the companies expense. The possible technicality is if the return wasn't agreed, so the company might claim they intended to collect the item or requested that the item be disposed of, but I think probably small claims courts would have common sense.

The real issue, and probably what the company is counting on - is it worth going even to small claims court for £8.50 plus expenses? And with even a small chance of losing?

Assuming that's not the case, Citizens Advice has a page here - they can give advice and forward the complaint to Trading Standards if they believe it's appropriate, or OP could report them to Trading Standards themselves.