NTA, he doesn't deserve brownies or an apology.

You can do whatever you want in the confort of your own home, it's your private space.

You could walk around naked, be having an orgy or doing a satanic ritual and it wouldn't be anybody's business (these are just examples taken to an extreme, I'm not condoning or promoting any behaviours, just stating that everyone should be able to do whatever they want in their own home).

If I noticed my neighbour walking around in underwear or naked, I'd look away as quickly as possible, would hope they didn't notice and think I'm a creep and would never speak of it to them.

You went above and beyond trying to compromise with his disconfort, he's the one in the wrong, not you OP.