Yachad-UK poll of British Jews shown to use biased sampling methods (RETRACTION)

Last week, the leftist British group Yachad released a poll of British Jews which received wide coverage. Its main findings are summarized by The Guardian:

75% of British Jews agree that “the expansion of settlements on the West Bank is a major obstacle to peace”, and 68% have a “sense of despair” whenever new expansion is approved.
73% believe Israel’s approach to peace is damaging its standing in the world.
71% see the two-state solution as the only way Israel can achieve peace.
72% reject the statement that “the Palestinians have no legitimate claim to a land of their own”.
62% support ceding territory to achieve peace, but that falls to 50% if withdrawal is seen as posing a risk to Israel’s security.
47% see the Israeli government as “constantly creating obstacles to avoid engaging in the peace process” (32% disagreed).

UK Media Watch noted how the questions in the poll themselves were biased, but there is a far more fundamental problem with the sampling methodology itself.

Over 50% of the respondents were chosen this way (page 50):

A common method for recruiting a sample from an inaccessible population is to identify a group of individuals from the population and invite them to recruit other members who will both complete the questionnaire and invite still others to do the same. ‘Snowballing’ methods of this kind are capable of achieving relatively high response rates because of the element of personal contact, but they run the risk of (i) recruiting an uncontrolled and potentially unrepresentative sample, and (ii) allowing abuse by vested interest groups who may submit multiple responses or circulate links to large numbers of people within that interest group.

In order to mitigate these risks, we developed a discriminative snowballing methodology with the following features:

(i) a group of 72 initial contacts (seeds) was selected by the research team and advisory group such that the group was roughly representative of the Jewish community as a whole with respect to synagogue affiliation, age and geographical location.

(ii) each seed was then asked to send invitations by email to between 10 and 40 of their Jewish contacts asking them to participate; each contact received a personal and unique code that could only be used once (phase 1).

(iii) the phase 1 recipients, in addition to being asked to complete the survey themselves, were provided with three additional unique codes and asked to send those to Jewish contacts of their own (phase 2). We limited the number to three to prevent blanket responding.

(iv) the phase 2 recipients were also asked to send links to up to three contacts using the unique links that were displayed on screen on completion of the survey (phase 3). This methodology generated 568 responses, of which 444 were generated at phase 1 and 124 at phases 2 and 3

So the advisory group chose the initial seed which then recruited the remainder of the 568.

Who was on the advisory group?

10 of them are listed in the paper, and Jonathan Hoffman identified nine of the ten as leftists.

It is of course likely that the leftist group would recruit like-minded friends, who would again do the same in their next pass. Meaning that fully 50% of the respondents to this poll are from a likely biased set.

Like J-Street in the US, Yachad commissioned this poll to confirm their own attitudes towards Israel and to find - or create the illusion of - widespread criticism of the Jewish state that mirrors their attitudes. Which is exactly what happened.

How many supporters of Likud do you think would be in the initial 72 people chosen by this leftist "advisory panel?"  Yet two thirds  of British Jews who expressed an opinion said before the Israeli elections,  that they would vote for Netanyahu if they were Israeli!

This is not just shoddy - this is deceptive.

UPDATE: As I mention in the comments, there is a very easy way to discover whether the people selected by the advisory group and then snowballed from there have the same opinions with the other half of those polled, whose methodology of selection seemed reasonably sound (at least in comparison.) Publish the results. If I am wrong, I will retract; if I am right and the results are significantly different, Yachad will issue an apology. Deal?

UPDATE 2: The report itself shows that the bias existed;

As predicted, the panel sample [145 people - EoZ] was the least close match; it under-represented younger Jews and to a lesser extent those with postgraduate degrees. These deviations would have shifted the findings in a hawkish direction. The snowball sample [568 people - EoZ] over-represented Jews with a left-leaning political stance and those with post-graduate qualifications; this would have produced a dovish bias. The DJN sample, as expected, was the closest match to the population profile; it slightly underrepresented members of central Orthodox synagogues and slightly over-represented older respondents; these two biases would have tended to neutralize one another in terms of any overall bias towards hawkish or dovish views.

So the pollsters admit that half the respondents held over-represented left leaning views!

The report then claims that it used weighting to eliminate the bias, but the weighting as to make the respondents fit better into the synagogue affiliation and age groups of the overall population.. But the left-leaning bias that they admit was not compensated for; they make the assumption that post-graduates will bring more post-graduates into the poll but not that people who are politically to the left would be far more likely to have friends who think the way they do.

And this footnote is interesting:

 Since it is impossible to determine which sample is the most representative, and to what extent each one departs from representativeness, there is no rational basis for weighting the samples relative to one another

Yet they made the stated assumption throughout that the DJN sample was the closest, demographically, to the Jewish population as a whole. That means that they are effectively giving four times the weight to the left-leaning snowball sample than to the (supposedly) right-leaning panel sample. 

Again, the only way to figure this out is to see the actual numbers and the specific formulas they used to supposedly reduce bias.

UPDATE 3: I am hearing that I am wrong - the data is similar between the snowballed sample and the rest. Will update when I get the details.

UPDATE 4/RETRACTION: Jonathan Hoffman contacted the pollsters and they took my challenge. The snowball and non-snowball halves had similar views on Israel, so in this case snowballing didn't majorly affect the responses except in a couple of specific questions.

I thank Stephen Miller for doing this analysis.

If I may interpret the differences between the groups, the "snowball" group seems to be more actively engaged in the issue of Israel than the other groups. They aren't more "hawkish" by any means but they are more supportive of Israel's right to exist as a Jewish country than the population at large and proud of Israel's achievements just as they are more supportive of a Palestinian state.

---

This blog may be a labor of love for me, but it takes a lot of effort, time and money. For over 11 years and over 22,000 articles I have been providing accurate, original news that would have remained unnoticed. I've written hundreds of scoops and sometimes my reporting ends up making a real difference. I appreciate any donations you can give to keep this blog going.

---