You should be able to fix that. Use it for toys. Warm blankets. Treats.
People on reddit seem to hate using a crate as punishment. I don't use it for punishment, but it can and has been his time out. It seems most people on reddit don't agrree with that. Ehh. There is a difference between punishment and using it correctly.
I still think you can potentially fix that issue, but I suppose but that depends on the dog. I still point at mine when he knows I'm upset. Maybe I should have a separate area for time outs. I've not had an issue with this with my last 4 dogs and I've always crate trained and sent him there for "tiny" time outs.
Best of luck. Maybe someone else could help more.