Rob Salmond at Polity has some changing thoughts on the subject of poll bias.
Whenever the media report on polls, they get accused of all manner of biases by the blinkered commentariat. Most of it is nonsense, of course. But some criticism of the polling methods themselves may have more merit that I first thought.
I have been keeping track of the polls in New Zealand for a few years now, mainly trying to get the rightindustry average level of support for each party, and figuring out ways to account for the biases in individual pollsters’ methods. Polity’s Poll of Polls is part of this work, as was the academic work I did uncovering bias in the old version of the Colmar Brunton poll.
But both Danyl at the Dim-Post and Gavin White at UMR have been looking at something more fundamental – is the polling industry biased as a whole? There is certainly a small enough number of competing firms in the market, and enough limitations on their work (e.g. the cost of calling mobiles) for bias to conceivably sneak in. For example, the best information I have seen suggests that the New Zealand population with access to a landline skew about 1.7% more towards National than the population as a whole. Calling a random selection of landlines, even if you do some weighting by demographics, can easily give you a result reflecting this bias.
Is this what is happening in New Zealand? Using quite different methods (but the same raw data), Danyl and Gavin both come to the conclusion that the New Zealand polling industry overstates National, modestly overstates the Greens, understates New Zealand First, and gets Labour about right.
I can see reasons other than poll bias why election results for the Greens may not reflect their polling. It could easily be that the young people who make up a lot of the Greens’ support base are just not very reliable when it comes to turning out in elections.
And the mechanism most people give for poll bias – the non-representative subset of the people who have landlines, should lead pollsters to overestimate New Zealand First with its older support base, not underestimate them. The underestimate of New Zealand First does not seem to fit the causal story people are telling.
But the fairly consistent results on National are harder to dismiss. In each of the last three elections we have been left staring at the chicken bones trying to explain how National’s support seemed to fall away in the dying days. A delayed Brethren effect? Collective panic at the thought of one party government? Maybe. Or maybe these are just the stories we tell ourselves in order to feel comfortable with the polls.
(Having said that, the polls in 2008 and 2011 did actually show a late-term decline for National, just not one big enough to account for the actual result. So the pollsters were not at all surprised in either case that National’s support was lower than its early campaign peak, but they were surprised just how far National had fallen.)
I was caught short in 2011 relying on the polling industry average by falsely projecting that National would get enough votes to govern alone, and that New Zealand First would not break 5%. I was wrong on both counts, and in exactly the direction that Danyl and Gavin would predict. Some might say that is all the evidence you need of bias.
Ultimately, however, I am not sold just yet on the notion of industry-level bias.
The studies Danyl and Gavin have done here can claim as many poll-level observations as they like, but in an important sense the number of elections is the number of observations, because that is the number of times you observe the true value of the things we are busy estimating. Which means right now we are operating on N equals three. Making conclusions on an N this small is risky.
While I’m not yelling “fire” at this stage, I do think there may be smoke. I did not think that three years ago.
If later this year we have to wave our hands at yet another inexplicable final week decline for National, and again marvel at Winston’s ability to sneak up on everyone at the last second, then that will substantially strengthen the claim of industry-wide bias.
If we find there is this bias down the track, then we will have to look very carefully at the way in which a slanted perception of the political reality today impacts on voters’ minds, usually through the journalistic filter. Do they unfairly nurture the reputations of over-estimated parties like National and the Greens, and unfairly tarnish New Zealand First. Does it have real consequences for those parties at the ballot box in an MMP environment? And, of so, is that political speech that should be protected, or protected against?
Powered by WPtouch Mobile Suite for WordPress