web analytics

Polity: X more thoughts on poll bias

Written By: - Date published: 11:36 am, February 26th, 2014 - 26 comments
Categories: national, nz first, polls - Tags: , ,

Rob Salmond at Polity has some changing thoughts on the subject of poll bias.

Whenever the media report on polls, they get accused of all manner of biases by the blinkered commentariat. Most of it is nonsense, of course. But some criticism of the polling methods themselves may have more merit that I first thought.

I have been keeping track of the polls in New Zealand for a few years now, mainly trying to get the rightindustry average level of support for each party, and figuring out ways to account for the biases in individual pollsters’ methods. Polity’s Poll of Polls is part of this work, as was the academic work I did uncovering bias in the old version of the Colmar Brunton poll.

But both Danyl at the Dim-Post and Gavin White at UMR have been looking at something more fundamental – is the polling industry biased as a whole? There is certainly a small enough number of competing firms in the market, and enough limitations on their work (e.g. the cost of calling mobiles) for bias to conceivably sneak in. For example, the best information I have seen suggests that the New Zealand population with access to a landline skew about 1.7% more towards National than the population as a whole. Calling a random selection of landlines, even if you do some weighting by demographics, can easily give you a result reflecting this bias.

Is this what is happening in New Zealand? Using quite different methods (but the same raw data), Danyl and Gavin both come to the conclusion that the New Zealand polling industry overstates National, modestly overstates the Greens, understates New Zealand First, and gets Labour about right.

I can see reasons other than poll bias why election results for the Greens may not reflect their polling. It could easily be that the young people who make up a lot of the Greens’ support base are just not very reliable when it comes to turning out in elections.

And the mechanism most people give for poll bias – the non-representative subset of the people who have landlines, should lead pollsters to overestimate New Zealand First with its older support base, not underestimate them. The underestimate of New Zealand First does not seem to fit the causal story people are telling.

But the fairly consistent results on National are harder to dismiss. In each of the last three elections we have been left staring at the chicken bones trying to explain how National’s support seemed to fall away in the dying days. A delayed Brethren effect? Collective panic at the thought of one party government? Maybe. Or maybe these are just the stories we tell ourselves in order to feel comfortable with the polls.

(Having said that, the polls in 2008 and 2011 did actually show a late-term decline for National, just not one big enough to account for the actual result. So the pollsters were not at all surprised in either case that National’s support was lower than its early campaign peak, but they were surprised just how far National had fallen.)

I was caught short in 2011 relying on the polling industry average by falsely projecting that National would get enough votes to govern alone, and that New Zealand First would not break 5%. I was wrong on both counts, and in exactly the direction that Danyl and Gavin would predict. Some might say that is all the evidence you need of bias.

Ultimately, however, I am not sold just yet on the notion of industry-level bias.

The studies Danyl and Gavin have done here can claim as many poll-level observations as they like, but in an important sense the number of elections is the number of observations, because that is the number of times you observe the true value of the things we are busy estimating. Which means right now we are operating on N equals three. Making conclusions on an N this small is risky.

While I’m not yelling “fire” at this stage, I do think there may be smoke. I did not think that three years ago.

If later this year we have to wave our hands at yet another inexplicable final week decline for National, and again marvel at Winston’s ability to sneak up on everyone at the last second, then that will substantially strengthen the claim of industry-wide bias.

If we find there is this bias down the track, then we will have to look very carefully at the way in which a slanted perception of the political reality today impacts on voters’ minds, usually through the journalistic filter. Do they unfairly nurture the reputations of over-estimated parties like National and the Greens, and unfairly tarnish New Zealand First. Does it have real consequences for those parties at the ballot box in an MMP environment? And, of so, is that political speech that should be protected, or protected against?


26 comments on “Polity: X more thoughts on poll bias”

  1. Blue 1

    Do they unfairly nurture the reputations of over-estimated parties like National and the Greens, and unfairly tarnish New Zealand First.

    The media don’t really care what the poll results are when it comes to how they represent certain parties. Witness two pieces in the Herald today focused on the 0% wonder.

    Their beef with NZ First is mostly journos having personal feuds with Winston Peters.

    If anything, the poll results which overestimate National simply give the media more of an excuse to downplay Labour, which they do anyway.

    • Crunchtime 1.1

      That doesn’t necessarily hold true when you are talking about a party with a large support base. If it looks (falsely) like a foregone conclusion that a party will win a majority vote, or close to, then it’s likely to demoralise and demotivate those who might vote against them.

      I’m not saying this is entirely to blame for the record-breaking 800,000 who didn’t vote last election, but I would suggest it’s one of many contributing factors.

    • Crunchtime 1.2

      …in addition, isn’t there the type of person who tends to vote with who seems to be the “winner” in order to give a “stable government”?

      If so then polls that show National as reasonably far ahead will have a gravitational effect, as it were, on future poll results favouring National, including the ultimate poll results of the election.

      Furthermore isn’t this a big reason why Labour has been polling so poorly for the past half decade, because it has been well publicised as being “divided” and therefore “unstable”? To be fair, Labour haven’t really done enough to dispel that notion.

  2. McFlock 2

    It also seems to me that there seems to be a bit of a bias towards the government of the day. No real data to back it up, though

    • Tracey 2.1

      I agree and think it is probably related tot he amoun to fmedia exposure a PM gets, compared to a LofO. Understandably the PM is a go to on many many things.

      I wonder, what would change, if anything, if published polls were outlawed in an election year?

    • Pasupial 2.2


      Polls do generally favour the incumbent. Most of the research on this is from a US setting, which I can’t be bothered churning through right now. This is the closest that I’ve been able to find to a NZ example thus far:


      You can see the higher digipoll numbers than votes achieved on election night for the incumbent National in 2011 (51% vs 47%) and Labour in 2005 (45% vs 41%). The transition years contradict this however; in 1999 Labour was favoured in the poll (40% vs 39%), in 2008 National (48% vs 45%). Plus in 2002 National was favoured in the digipoll (23% vs 21%).

      I really need to get more NZ data on this though. The digipoll numbers seem to be averaged across the election year. Online; Roy Morgan only goes back to January 2012, and; Colmar Brunton to February 2011.

    • JK 2.3

      that didn’t happen with Helen Clark in her final term ….. media were very biased against her.

  3. Lanthanide 3

    “And the mechanism most people give for poll bias – the non-representative subset of the people who have landlines, should lead pollsters to overestimate New Zealand First with its older support base, not underestimate them. The underestimate of New Zealand First does not seem to fit the causal story people are telling.”

    Do elderly living in rest homes have their own individual phone lines? I’d suggest not all of them would.

    • Tracey 3.1

      If it is like other types of residential care there is a main line with a voice message allowing you to input the extension you require or else wait to get the reception?

      • Lanthanide 3.1.1

        Right, in which case there would only be 1 number in the phone book which would cover all residents, rather than 1 line per resident.

        So that would help to explain some of why NZ1st is under-represented.

  4. Ant 4

    How many people have their numbers listed these days anyway?

    • Pasupial 4.1


      They don’t need a phone listing – they’ve got computers configured to generate random 7 digit numbers (plus area prefix). Or in Colmer Brunton speak; “Nationwide random digit dialling of landline telephones using stratified random probability sampling”.

      • Ant 4.1.1

        Cheers for that, for some reason I had it in my head that calling like that was legislated against and they had to use public information.

      • jaymam 4.1.2

        The random phone numbers are driven by a table for each telephone exchange that shows the telephone numbers actually in use. If those tables have not been updated since the program was supplied (by me!) to the pollster, the newly added numbers and new exchanges will not be polled. Those will tend to be in the outer cheaper suburbs where they are more likely to be lefties.

        • Pasupial

          Cheers for that info jaymam. Any idea how often those number tables are updated in the company you’ve had dealings with? [to remained unnamed obviously]

          • jaymam

            The director that I gave the program to ended up suing the programmer for a lot of money, for a different program. So I doubt that the tables have ever been updated. It is very possible they use a different program now, but they possibly think that nobody would notice if they don’t call all the new numbers.That director has now fallen out with the company that bears her name and now does some polling on her own, which has been widely criticised.

  5. fambo 5

    I wonder how difficult it would be for external agencies to manipulate polls to destabilise a government. They could, for instance, pump large sums of money into setting up their own polling company. Or ferret their operatives into positions of importance in extant polling companies.

  6. rain33 6

    Polls Shmolls, who can forget the embarrassing fail of the right wing, namely Dick Morris and Dean Chambers who had Romney winning the 2012 US election right up until the night of the election? Their methods of ‘unskewing’ the polls sounded quite justified, by re-weighting the sample to match what they believed the electorate would look like in terms of party identification, but as history has proven were completely off the mark.

    I would highly recommend the book The Signal and the Noise: Why Most Predictions Fail – but Some Don’t – by Nate Silver the ‘god of polling’ and American statistician, who in the 2012 US Presidential election correctly predicted the outcome of all 50 States and the District of Columbia, that same year his U.S. Senate race predictions were correct in 31 of 33 States, in 2008 he got 35/35

    Individual polls are an indication of a moment in time. However, when polls start to indicate a trend, that is worth paying attention to. That should be of a concern to Labour. The debate regarding cellphones versus landlines etc was discussed ad-nauseum in the US but became a mute point. For all we know people who have landlines are more likely to turn up and vote at the election booth than people who don’t have landlines. You will get giddy trying to work out all the variables. Instead Labour should be focusing on getting a cohesive message out to the voting public.

  7. Whatever next 7

    Do our polls use the same polling techniques as Nate Silver?

    • Lanthanide 7.1

      Nate Silver didn’t perform any polling himself, he just aggregated and number-crunched the polls that others did.

      In the process he uncovered statistical proof that one of the polling companies was deliberately biasing their polls towards the republicans – they shut up shop after that IIRC.

    • Tamati 7.2

      As said above, Silver doesn’t poll. Silver uses what’s called Bayesian statistics. It’s a totally different way of looking at sampling and prediction and is very trendy ATM.

      Unfortunately it would be very hard to replicate here because we too few polls and under MMP you don’t have an absolute ‘winner’ like you do in FPP,

  8. swordfish 8

    Here are the Poll averages from previous Election years for both (1) February (to help anticipate 2014 Election result from current (February) polling) and (2) the 4-week period immediately before the particular Election (with Election Result comparisons):


    Nat (Feb average) 53% / Election 47% (minus 6 points)
    Nat (Final 4-weeks average) 52% / Election 47% (minus 5 points)

    Lab (Feb average) 33% / Election 27% (minus 6 points)
    Lab (Final 4-weeks average) 28% / Election 27% (minus 1 point)

    Green (Feb average) 8% / Election 11% (plus 3 points)
    Green (Final 4-weeks average) 12% / Election 11% (minus 1 point)

    NZF (Feb average) 3% / Election 7% (plus 4 points)
    NZF (Final 4-weeks average) 3% / Election 7% (plus 4 points)


    Nat (Feb average) 53% / Election 45% (minus 8 points)
    Nat (Final 4-weeks average) 47% / Election 45% (minus 2 points)

    Lab (Feb average) 34% / Election 34% (Equal)
    Lab (Final 4-weeks average) 35% / Election 34% (minus 1 point)

    Green (Feb average) 6% / Election 7% (plus 1 point)
    Green (Final 4-week average) 8% / Election 7% (minus 1 point)

    NZF (Feb average) 3% / Election 4% (plus 1 point)
    NZF (Final 4-weeks average) 3% / Election 4% (plus 1 point)


    Nat (Feb average) 37% / Election 39% (plus 2 points)
    Nat (Final 4-weeks average) 40% / Election 39% (minus 1 point)

    Lab (Feb average) 45% / Election 41% (minus 4 points)
    Lab (Final 4-weeks average) 40% / Election 41% (plus 1 point)

    Green (Feb average) 5% / Election 5% (Equal)
    Green (Final 4-weeks average) 6% / Election 5% (minus 1 point)

    NZF (Feb average) 5% / Election 6% (plus 1 point)
    NZF (Final 4-weeks average) 6% / Election 6% (Equal)

    Need I say more ? (I need not).

  9. Crunchtime 9

    I was trying to find some info about opinion polls prior to the 2002 election but can’t seem to find any on the internet. There’s a lot of info on opinion polls for 2005 but not the previous…

    All I could find was this story about how polls were pointing to an outright majority, but people “withdrew their support” to deny them this. Clearly, National didn’t benefit from this either, nearly slipping below 20% support, which I believe is far lower than Labour’s support has ever fallen.

    I also found some interesting food for thought: the “Electoral (Public Opinion Polls) Amendment Bill” 2002. This is to outlaw any publicity whatsoever 28 days prior to the election. Because OPINION POLLS INFLUENCE RESULTS. Interesting huh… As if banning them for 28 days is going to make any difference!

    Election results 2002:

    Labour 41.3%

    National 20.9%

    NZ First 10.4%

    Act 7.1%

    Green 7%

    United Future 6.7%

    Progressive 1.7%

    Possibly the most even spread of votes in NZ election history?

    It would be interesting to see what opinion polls were doing leading up to this result.

Recent Comments

Recent Posts


  • A charge on plastic bags – debunking some myths
    The launch of my Members’ Bill last week, which would introduce a 15 cent charge on single-use plastic bags at the check-out, has generated a lot of comment on mainstream and social media. From The Paul Henry Show at the ...
    GreensBy Denise Roche
    1 day ago
  • National’s $1trillion property sandcastle
    The National government's failure to fix the housing crisis has seen the ballooning and unsustainable property market touch the $1 trillion mark, Labour’s Finance spokesperson Grant Robertson says. "Labour wants an economy that creates high wage work that is based ...
    1 day ago
  • Government failure on housing crisis drives Reserve Bank to add tools
    If the Government was delivering a comprehensive plan to fix the housing crisis, it is unlikely that the Reserve Bank would be continuing to pursue debt to income limits for lending for housing, says Labour’s Finance spokesperson Grant Robertson. ...
    2 days ago
  • International embarrassment for NZ likely over National’s failure to protect Maui dolphin
    New Zealanders who care about Maui dolphin should prepare to feel embarrassed: the Government is about to be put to shame on the international stage for its lack of action to protect Maui’s dolphin. The International Whaling Commissions’ 66th Biennial ...
    GreensBy Eugenie Sage
    4 days ago
  • Why don’t we spend $1b to keep people out of jail, rather than spending it on keeping them in?
    Earlier this week, Corrections Minister Judith Collins announced the government’s ‘solution’ to our burgeoning prison population. It seems that most, if not all, of Bill English’s hard-won surplus is going to disappear into another round of prison-building.  That must be ...
    GreensBy David Clendon
    6 days ago
  • PKE Ship Sent Packing – Not Too Soon
    It is appropriate that the palm kernel expeller (PKE) ship off Tauranga has been sent packing. For weeks I have been saying this ship needed to be sent away, but it seems as if MPI has been trying to find ...
    GreensBy Steffan Browning
    6 days ago
  • Do you #LoveSnow?
    I was a lucky kid. When I was about five or six my mum and auntie took me up to Whakapapa on Mt Ruapehu and taught me to ski. As a young kid I thought there was no bigger ...
    GreensBy James Shaw
    6 days ago
  • Awa Kairangi/Hutt River – Swimmable?
    On Thursday night I hosted a great swimmable rivers meeting organised by the local Greens in Heretaunga (Hutt Valley). It was great to see about 70 people attend and engage in the topic. We were welcomed by Te Atiawa representative ...
    GreensBy Catherine Delahunty
    6 days ago
  • September benefit figures disappointing
    The Government is out of touch with the reality that fewer people are going off the benefit and into employment or study, says Labour’s Social Development spokesperson Carmel Sepuloni.  “The quarterly benefit numbers for September are concerning. They show that ...
    6 days ago
  • MFAT officials refuse to back Prime Minister on Saudi sheep claims
    An Ombudsman’s interim decision released about the existence or otherwise of legal advice on the multimillion dollar Saudi sheep deal shows MFAT has failed to back up the Prime Minister’s claims on the matter, says Labour MP David Parker. “The ...
    6 days ago
  • Barry Coates on his first weeks in Parliament
    Week one in Parliament has been quite an occasion. I would like to share the experience. I had given up on the prospect of getting into Parliament before the election and had been enjoying the diverse work I was doing ...
    GreensBy Barry Coates
    6 days ago
  • Nats still planning to take Housing NZ dividend
    Housing New Zealand’s Statement of Performance Expectations shows that the National Government intends to pocket $237m from Housing New Zealand this year including a $54m “surplus distribution”, despite promises that dividends would stop, says Labour’s Housing spokesperson Phil Twyford. “After ...
    1 week ago
  • Parliament must restore democracy for Ecan
    Parliament has a chance to return full democracy to Canterbury with the drawing of a member’s bill that would replace the Government’s appointed commissioners with democratically elected councillors, says Labour’s Canterbury Spokesperson Megan Woods. “In 2010, the Government stripped Cantabrians ...
    1 week ago
  • Police struggle to hold the line in Northland
    Labour’s promise of a thousand extra police will go a long way to calming the fears of people in the North, says the MP for Te Tai Tokerau Kelvin Davis.  “Police are talking about the Northland towns of Kaitaia and ...
    1 week ago
  • Vote Sooty Shearwater/Tītī for Bird of the Year
    Sooty shearwater (Puffinus griseus) are amazing and deserve your vote in Forest and Bird’s Bird of the Year competition.  They make one of the longest known bird migrations, flying an annual round trip of 64,000 kms across the entire Pacific ...
    GreensBy Eugenie Sage
    1 week ago
  • Urgent action on agriculture emissions needed
    Immediate action is required to curb agricultural emissions is the loud and clear message from Climate change & agriculture: Understanding the biological greenhouse gases report released today by the Parliamentary Commissioner for the Environment, says Labour’s Climate Change spokesperson Megan ...
    1 week ago
  • Super Fund climate change approach a good start
    Labour Finance Spokesperson Grant Robertson and Climate Change Spokesperson Dr Megan Woods have welcomed the adoption of a climate change investment strategy by the New Zealand Super Fund. “This is a good start. It is a welcome development that the Super ...
    1 week ago
  • Energy use going in the wrong direction
    New data out this week from Statistics NZ paints a concerning picture of energy use across the economy under this National Government. You won’t be surprised to hear that there is some seriously worrying information here about how dirty our ...
    GreensBy Gareth Hughes
    1 week ago
  • Raising the age the right thing to do
    The announcement today that the Government will leave the door open for young people leaving state care still means there is a lot of work to do, says Labour's Spokesperson for Children, Jacinda Ardern "The Government indicated some time ago ...
    1 week ago
  • Junior Doctors go on Strike
    Thousands of junior doctors took strike action for 24 hours this week for better working conditions and safer working hours.  The Green Party supports their cause, and particularly their claims to reduce the number of days worked from up to ...
    GreensBy Denise Roche
    1 week ago
  • Coleman plays down the plight of junior doctors
    Junior doctors are crucial to our health services and the industrial action that continues tomorrow shows how desperately the Government has underfunded health, says Labour’s Health spokesperson Annette King.  “Jonathan Coleman’s claim that he has not seen objective evidence of ...
    1 week ago
  • Inflation piles pressure on National and Reserve Bank
    While many households will welcome the low inflation figures announced today, they highlight serious questions for both the National government and the Reserve Bank, Labour’s  Finance Spokesperson Grant Robertson said.  "While low inflation will be welcomed by many, the ...
    1 week ago
  • Officials warned Nat’s $1b infrastructure fund ineffective and rushed
    Treasury papers show the Government rushed out an infrastructure announcement officials told them risked making no significant difference to housing supply, says Labour’s housing spokesperson Phil Twyford.  “Like so much of National’s housing policy, this was another poll-driven PR initiative ...
    1 week ago
  • More cops needed to tackle P
    New Police statistics obtained in Written Questions show John Key is losing his War on P, highlighting the need for more Police, says Opposition Leader Andrew Little.  “New Zealanders expect serious action on P but today’s hodgepodge of half-measures won’t ...
    1 week ago
  • Strengthening our relationship with the Rātana movement
    It was a privilege to visit Rātana Pā last week with fellow Greens’ Co-leader James Shaw, our Māori Caucus and senior staff to meet with the leaders of te iwi mōrehu, to strengthen the ties between the Green Party and ...
    GreensBy Metiria Turei
    1 week ago
  • MBIE docs show country needs KiwiBuild, not Key’s pretend “building boom”
    John Key’s spin that New Zealand is in a building boom does not change the massive shortfall in building construction as new MBIE papers reveal, says Labour Party housing spokesperson Phil Twyford.  “We can fix the housing crisis, by the ...
    1 week ago
  • 1 in 7 Akl houses now going to big property speculators
    Speculators are running riot in the Auckland housing market making life tougher for first home buyers, says Labour’s housing spokesperson Phil Twyford.  Newly released data from Core Logic shows a 40 per cent increase in the share of house sales ...
    2 weeks ago
  • Disconnected thinking dirties the water
    Iain Rabbitts’ belief that drinking water quality, charging for water use and the land use that leads to water quality degradation should be treated separately is part of the problem we have right now in this country. The connection is ...
    GreensBy Catherine Delahunty
    2 weeks ago
  • Report back from Hands Off Our Tamariki hui
    This week I attended a hui in Otaki organised by Hands Off Our Tamariki about the proposed reforms to the Child Young Persons and their Families Act. Moana Jackson and Paora Moyle spoke.  They expressed deep, profound concern about the proposed ...
    GreensBy Jan Logie
    2 weeks ago
  • Labour mourns passing of Helen Kelly
    Helen Kelly was a passionate advocate for working New Zealanders and for a safe and decent working life, Leader of the Opposition Andrew Little says.  “Helen Kelly spent her adult life fighting for the right of every working person to ...
    2 weeks ago
  • National’s visionless immigration policy
    National’s recent immigration announcement is a continuation of the visionless approach to government that it has displayed in the last three terms. Rather than using the levers of government to implement a sustainable immigration policy that benefits new and current ...
    GreensBy Denise Roche
    2 weeks ago
  • Andrew Little: Speech to the Police Association Conference 2016
    Police Association delegates, Association life members and staff, representatives from overseas jurisdictions. Thank you for inviting me here today. The Police Association has become a strong and respected voice for Police officers and for policing in New Zealand. There is ...
    2 weeks ago
  • 1,000 more police for safer communities
    Labour will fund an extra 1,000 Police in its first term to tackle the rising rate of crime, says Leader of the Opposition Andrew Little. “Labour will put more cops on the beat to keep our communities safe. ...
    2 weeks ago
  • Call for all-party round table on homelessness
    Labour is calling on the Government to take part in a roundtable meeting to hammer out a cross-party agreement on ending homelessness.  Labour’s housing spokesperson Phil Twyford said the country wanted positive solutions to homelessness, and wanted the political parties ...
    2 weeks ago
  • Seclusion rooms in schools
    Schools are undoubtedly stretched and underfunded to cope with students with high learning support needs. But this cannot justify the use of rooms (or cupboards) as spaces to forcibly isolate children. It has emerged via media that this practice continues ...
    GreensBy Catherine Delahunty
    2 weeks ago
  • Public should get a say on new Waikato power station
    I had an opinion piece published in the Waikato Times about a controversial proposal to build a new gas-fired power station. It’s not on their website yet, so here it is: If you think the public would get a say ...
    GreensBy Gareth Hughes
    2 weeks ago
  • MSD and their investment approach
    The Government talks about investment but there is no investment. It is not investment if it isn’t over the whole of life and if there is no new money  — Shamubeel Eaqub   Investment sounds like adequate resourcing but this ...
    GreensBy Jan Logie
    2 weeks ago
  • Certainty needed for community services
    A couple of months ago I was at a seminar where three community organisations were presenting. Two of the three presenters were waiting to find out if their organisation would get a contract renewed with MSD. Not knowing if their ...
    GreensBy Jan Logie
    2 weeks ago
  • Domestic Violence – some advice for the media
    For the purpose of this piece, I’m going to use Domestic Violence (DV) as a proxy for intimate partner violence. DV is not isolated to physical abuse in a relationship between people with the same power. DV is a pattern of ...
    GreensBy Jan Logie
    3 weeks ago
  • Leroy’s New Paw Prints
    Leroy, an Auckland great dane recently received a new 3D printed bionic leg after cancer was discovered. I think this is a fantastic story and highlights the real potential of additive manufacturing, or 3D printing Leroy’s prosthetic was printed in titanium and was ...
    GreensBy Gareth Hughes
    3 weeks ago