In short, no.
This is not a question that can be settled with data. Instead, I'm going to look at reasons why people might want a reduction in immigration other than out of a petty dislike of foreigners.
Let's start with Tyler Cowen and what still seems to me to be the most perceptive and interesting analysis of the Brexit vote. it's about England. Here are some excerpts, but you should read the whole thing:
"As I interpret what happened, ultimately the vote was about preserving the English nation, and yes I use those last two italicized words deliberately; reread Fintan O’Toole. Go back and read English history. For centuries, England has been filled with English people, plus some others from nearby regions. Go visit Norfolk and also stop in Great Yarmouth, once described by Charles Dickens as “…the finest place in the universe,” and which, for whatever decline it may have experienced, still looks and feels like England. London does not.
Much has been made of the supposed paradox that opposition to immigration is highest where the number of immigrants is lowest. Yes, some of that is the racism and xenophobia of less cosmopolitan areas, but it would be a big mistake to dismiss it as such or even to mainly frame it as such. Most of all it is an endowment effect. Those are the regions which best remember — and indeed still live — some earlier notion of what England was like. And they wish to hold on to that, albeit with the possibility of continuing evolution along mostly English lines.
One way to understand the English vote is to compare it to other areas, especially with regard to immigration. If you read Frank Fukuyama, he correctly portrays Japan and Denmark, as, along with England, being the two other truly developed, mature nation states in earlier times, well before the Industrial Revolution. And what do we see about these countries? Relative to their other demographics, they are especially opposed to very high levels of immigration. England, in a sense, was the region “out on a limb,” when it comes to taking in foreigners, and now it has decided to pull back and be more like Denmark and Japan.
The regularity here is that the coherent, longstanding nation states are most protective of their core identities. Should that come as a huge surprise? The contrast with Belgium, where I am writing this, is noteworthy. The actual practical problems with immigration are much greater here in Brussels, but the country is much further from “doing anything about it,” whether prudently or not, and indeed to this day Belgium is not actually a mature nation-state and it may splinter yet. That England did something is one reflection of the fact that England is a better-run region than Belgium, even if you feel as I do that the vote was a big mistake."The English, like the Danish and the Japanese, think of themselves as distinct from their neighbours. There's decent historical evidence to support that. Read Robert Tombs (the French original, which introduced me to the word "rafistolage", is here), especially his The English and Their History, a book I highly recommend (again). Wishing to preserve that distinctiveness through restricting immigration is not inherently wrong.
Coming at things a little differently, here's David Goodhart, saying that wishing to reduce immigration is perfectly legitimate. (I liked this: "Britain became a mass immigration society much as it became an imperial one—in a fit of absence of mind.")
I can go further: even where the resistance to immigrants is expressed in terms that imply a dislike of them, there is good reason to think that that is not the true motivation for that resistance. That sounds rather convoluted so I'll explain by telling you a little story.
Imagine a charming little village, perhaps an old fishing village or perhaps a picture-postcard Cotswold village. It is a comfortably-off place, with a fair number of old local families, a few newcomers and a scattering of holiday homes and weekend cottages. One inhabitant is a gay Australian called X who moved there from London a few years back. He's now a pillar of the community and universally popular. Everyone lives together very happily.
One summer, X decides to invite all his old friends to come and stay. They love it. Next summer, they comes back - with all their friends too. The summer after that, all their friends' friends come too and the village is over-run with visitors. It gets worse year after year.
I said it would be one story, but really it's three.
- Story (A): the visitors who come back year after year are Australians. For a couple of weeks each year, Australian Rules Football fills the local park, the pubs all serve Australian lager (by popular demand) and Australian accents dominate the streets. "For two weeks each year, this sleepy English village becomes a suburb of Sydney!" the Herald Sun writes.
- Story (B): the visitors are gay. For a couple of weeks each year, the local park becomes an unofficial Gay Pride festival, the pubs all hoist rainbow flags and gay displays of affection dominate the streets. "For two weeks each year, this sleepy English village becomes Old Compton Street on Sea!" the Pink News writes.
- Story (C): the visitors are all Londoners with children. For a couple of weeks each year, the local park is turned into a farmers' market, the pubs serve skinny lattes and giant buggies filled with children called Milo dominate the streets. "For two weeks each year, this sleepy English village becomes Chelsea in the Cotswolds!" the Evening Standard Magazine writes.
Now in each case, the locals do not like the incomers. They want their park, pubs and streets to be the way they once were. They wish the visitors would stop coming. They get frustrated and angry. In story (A), they frequently make anti-Australian jokes among themselves, often within earshot of the visitors; in story (B) they occasionally stoop to derogatory comments about gay people when they consider themselves entirely among friends; while in story (C) they have no hesitation in complaining loudly about all these bloody ridiculous people with their obsession with organic food and 4x4s.
I don't think we should properly regard the locals as xenophobic in story (A), nor homophobic in story (B). Their frustration just comes out that way. They had nothing against X, you recall. There's no bad word for what they are doing in story (C), so they do it more openly, but it's essentially the same frustration as in the other stories - a dislike of a large influx of different people.
I have no proof that that kind of mental process explains antipathy to immigration in the context of the Brexit vote, but it seems plausible to me. I would guess that a massive migration to Boston of British but notably distinct people old Etonians, say) would produce much the same response as the influx of east Europeans has done.
There are ways of testing this. Do the people of Leave-voting areas go out of their way to find immigrants to pick fights with them? Do they object to immigrants coming to big cities like London? Do they consider Polish people to be inferior to British people in some respect? Do local men suspect immigrant men of having insatiable sexual appetites and an eye on the local women? How is English-Polish inter-marriage regarded? My strong suspicion is that the answers to those questions would show that modern hostility to EU immigration is not motivated by the same sorts of unpleasant visceral racist feelings that motivated earlier hostility to immigrants from, say, the West Indies.
Finally, here is some evidence from newspaper articles. First, the Daily Mail "I am also surprised by the lack of animosity. True, there is palpable anger among native Bostonians that their town has become a Baltic satellite in next to no time. // But it’s not an anger directed at the Eastern Europeans. It’s aimed squarely at the Government and the EU. ... While organising protests against immigration, [a UKIP recruit called Bob] also sits on the Boston Good Relations Group, along with representatives of the migrant community, such as Ziedonis, whom he likes and respects. // ‘We don’t mind the migrants. I blame the damned politicians who have created this problem,’ he says."
[This paragraph is an update.] Second, the Economist: "remember that [Britain] is not at present a racist or intolerant country. Britons (including old, white, working-class Britons) might dislike immigration, but they tend also to dislike racism and discrimination. Eurobarometer, which obsessively polls Europeans on their prejudices, consistently finds that Britons are unusually relaxed about the idea of having a non-white political leader or non-white co-workers. The most recent poll, in 2015, found that only Swedes were as calm as Britons at the prospect of one of their children dating a Muslim." Listen also to this at 2:20 - 2:40: interviewees expressed no antipathy to foreigners, but a concern for control and identity.
Third, the Guardian, which spends some time flushing out some genuine racists but also has this: "She and her mother [who married a man from Trinidad and Tobago] do not strike me as essentially motivated by prejudice; rather people faced with an insecure, unpredictable world whose governing logic they can no longer abide, least of all politically." Well, quite. That was in March 2014: they've had a chance to try to make the world more secure and predictable since then, and I bet they took it.