He also got beat up by poor thugs, bullied by his poor clown coworkers, and not listened to by his underfunded therapist. In the end he had no parents at all, his mom adopted him as far as he knew. Rich, poor, middle class, nobody even looks at him as he walks by. Murray and Wayne didn't abuse their privilege, in his mind they abused their roles as father-figures and denied him their recognition. He isn't oppressed by an identifiable enemy in a material sense, just rebuked at every turn by people from whom he wants acknowledgement.
Nobody cares about him, his jokes, or his illness by the end of the movie, so he stops caring too. For me it seems pretty 'believable'.