Not sure, it sounds good and it's good for political soundbites. For sure, they have access to more "tools" to optimise their tax affairs. And those tools that are deemed to aggressive get shut down and punished by HMRC.
I think at quite a few levels of society, people avoid tax. Like builders who give people a "cash price" without VAT, cash restaurants, home service workers (cleaners, hairdressers, nails, babysitters, tutors, music teachers), landlords that don't fill in tax returns...
If someone is on PAYE earning £100k+, they are certainly paying more tax by any metric than cash traders. Also, don't forget that the income allowance goes after a certain income, the pension annual allowance also drops after a certain income.
I'm not against the rich paying their fair share, but it's not clear what that means, if it's not already the case and if it's not, how to achieve that.