I agree. Speaking from the viewpoint of someone in the UK, who is generally rather pro-immigration (within obvious limits, which is kind of your point if I'm correct), and voted to stay in the EU, a lot of people made the argument after the referendum that 'It was the areas with the most EU migrants that voted remain, and the ones with the areas with the least that voted to leave - they don't understand the benefits of it' (sidenote: non-UK born EU citizens who were not UK citizens (i.e. most of them) could not vote, so that's not a factor).

Now, this is partly true. However, the most heavily leave-voting areas in the country were largely concentrated about halfway up the East Coast of the country. In this area, there has always been a lot of opportunity for the less well-off in nearby countries, to come over in the summer, and work in industries such as fishing and fruit picking. The local people didn't lose jobs because of it, and the migrants were generally friendly. After relaxed EU migration rules, this tradition continued - however, most of the migrants chose to stay full time, rather than summer only. This put a lot of strain on public services, which was a large factor in people in the area voting to leave the EU.

Large scale migration purely from one part of the world to one part of the UK has always been a bit of a problem as well (Go to certain neighbourhoods in Bradford or Birmingham and the number of people who can trace their roots to the city of Mirpur in Pakistan is huge).