Switzerland is a part of the EU social charter for the free movement of labour to and from its borders. You still need a work permit and residence which is arranged beforehand by your employer. Germany and France no longer require work permits.
Canada, America, New Zealand and Australia not an EU issue obviously, I use the Skill shortage visa (albeit temporary) I am merely pointing out that I haven't personally experienced issues found in the United Kingdom in these countries as the cost of living is more in tune with their national wage, thus higher quality of life.
The point people are making is about immigration rates, if you look at the list of countries by number of immigrants you will see the majority of countries I mentioned above are ranked just as highly as those in the United Kingdom.
People in the UK aren't losing out because the UK offers lower pay to immigrant labour, people are losing out because the UK aren't keeping wages in tune with costs of living.
This past summer I was in Canada earning 11.50 an our, that's about $2 over the minimum wage. I lived in an expensive area due to it being a national park but payed $400 monthly in rent, I ate out frequently, took a few holidays to the US, bought ski gear, and frivolous items and still had enough cash to book my flight out with a few thousand savings by the time I moved on to Switzerland.
Last time I worked in the UK I experienced the exact opposite, my friends and family here (outside of the main cities) seem to spend nothing but have nothing. Garbage. Honestly the price of everything here is ludicrous.
Let's put it in another way, lets say I go out for a dominoes Pizza.
In the UK i'm earning 7.20 an hour, I'll grab a 12" pepperoni and it will set me back 14.99 (2 hours work)
Switzerland, 22 francs an hour it will set me back 14 francs (Less than 1 hour work)
Australia, 19 Bucks an hour, I can grab a pizza for $5 (15 minutes work)
Canada, 11.00 an hour, Brooklyn 14" costs $10. (one hour)
Now I know you're thinking i'm on crack and what's my obsession with Pizza but I'm simply pointing out the disparity between costs and income. Apply this to rent, phone credit, transportation and the story is the same across the board. The UK is a rip off and the people at the bottom are the ones suffering for it.