There is actual proof that Diamond City has treated Ghouls far worse then the Brotherhood has. I'd say physically, violently kicking out what was likely dozens of non feral ghouls from a town they probably helped build and lived in for over 100 years is much worse then snide remarks some of the Brotherhood members say towards them.

Hell the Brotherhood doesn't even do anything about one of their Knights/Paladins/Sentinels running around with a pet Mutant or non hostile Synths/Ghouls as they are true to their word they don't go after non hostiles. We can talk them down from killing the only two assassination targets Danse and Virgil, especially Virgil since he would be on top of their shit list for over a decade of FEV research.