web analytics

Polity: X more thoughts on poll bias

Written By: - Date published: 11:36 am, February 26th, 2014 - 26 comments
Categories: national, nz first, polls - Tags: , ,

Rob Salmond at Polity has some changing thoughts on the subject of poll bias.

Whenever the media report on polls, they get accused of all manner of biases by the blinkered commentariat. Most of it is nonsense, of course. But some criticism of the polling methods themselves may have more merit that I first thought.

I have been keeping track of the polls in New Zealand for a few years now, mainly trying to get the rightindustry average level of support for each party, and figuring out ways to account for the biases in individual pollsters’ methods. Polity’s Poll of Polls is part of this work, as was the academic work I did uncovering bias in the old version of the Colmar Brunton poll.

But both Danyl at the Dim-Post and Gavin White at UMR have been looking at something more fundamental – is the polling industry biased as a whole? There is certainly a small enough number of competing firms in the market, and enough limitations on their work (e.g. the cost of calling mobiles) for bias to conceivably sneak in. For example, the best information I have seen suggests that the New Zealand population with access to a landline skew about 1.7% more towards National than the population as a whole. Calling a random selection of landlines, even if you do some weighting by demographics, can easily give you a result reflecting this bias.

Is this what is happening in New Zealand? Using quite different methods (but the same raw data), Danyl and Gavin both come to the conclusion that the New Zealand polling industry overstates National, modestly overstates the Greens, understates New Zealand First, and gets Labour about right.

I can see reasons other than poll bias why election results for the Greens may not reflect their polling. It could easily be that the young people who make up a lot of the Greens’ support base are just not very reliable when it comes to turning out in elections.

And the mechanism most people give for poll bias – the non-representative subset of the people who have landlines, should lead pollsters to overestimate New Zealand First with its older support base, not underestimate them. The underestimate of New Zealand First does not seem to fit the causal story people are telling.

But the fairly consistent results on National are harder to dismiss. In each of the last three elections we have been left staring at the chicken bones trying to explain how National’s support seemed to fall away in the dying days. A delayed Brethren effect? Collective panic at the thought of one party government? Maybe. Or maybe these are just the stories we tell ourselves in order to feel comfortable with the polls.

(Having said that, the polls in 2008 and 2011 did actually show a late-term decline for National, just not one big enough to account for the actual result. So the pollsters were not at all surprised in either case that National’s support was lower than its early campaign peak, but they were surprised just how far National had fallen.)

I was caught short in 2011 relying on the polling industry average by falsely projecting that National would get enough votes to govern alone, and that New Zealand First would not break 5%. I was wrong on both counts, and in exactly the direction that Danyl and Gavin would predict. Some might say that is all the evidence you need of bias.

Ultimately, however, I am not sold just yet on the notion of industry-level bias.

The studies Danyl and Gavin have done here can claim as many poll-level observations as they like, but in an important sense the number of elections is the number of observations, because that is the number of times you observe the true value of the things we are busy estimating. Which means right now we are operating on N equals three. Making conclusions on an N this small is risky.

While I’m not yelling “fire” at this stage, I do think there may be smoke. I did not think that three years ago.

If later this year we have to wave our hands at yet another inexplicable final week decline for National, and again marvel at Winston’s ability to sneak up on everyone at the last second, then that will substantially strengthen the claim of industry-wide bias.

If we find there is this bias down the track, then we will have to look very carefully at the way in which a slanted perception of the political reality today impacts on voters’ minds, usually through the journalistic filter. Do they unfairly nurture the reputations of over-estimated parties like National and the Greens, and unfairly tarnish New Zealand First. Does it have real consequences for those parties at the ballot box in an MMP environment? And, of so, is that political speech that should be protected, or protected against?

26 comments on “Polity: X more thoughts on poll bias ”

  1. Blue 1

    Do they unfairly nurture the reputations of over-estimated parties like National and the Greens, and unfairly tarnish New Zealand First.

    The media don’t really care what the poll results are when it comes to how they represent certain parties. Witness two pieces in the Herald today focused on the 0% wonder.

    Their beef with NZ First is mostly journos having personal feuds with Winston Peters.

    If anything, the poll results which overestimate National simply give the media more of an excuse to downplay Labour, which they do anyway.

    • Crunchtime 1.1

      That doesn’t necessarily hold true when you are talking about a party with a large support base. If it looks (falsely) like a foregone conclusion that a party will win a majority vote, or close to, then it’s likely to demoralise and demotivate those who might vote against them.

      I’m not saying this is entirely to blame for the record-breaking 800,000 who didn’t vote last election, but I would suggest it’s one of many contributing factors.

    • Crunchtime 1.2

      …in addition, isn’t there the type of person who tends to vote with who seems to be the “winner” in order to give a “stable government”?

      If so then polls that show National as reasonably far ahead will have a gravitational effect, as it were, on future poll results favouring National, including the ultimate poll results of the election.

      Furthermore isn’t this a big reason why Labour has been polling so poorly for the past half decade, because it has been well publicised as being “divided” and therefore “unstable”? To be fair, Labour haven’t really done enough to dispel that notion.

  2. McFlock 2

    It also seems to me that there seems to be a bit of a bias towards the government of the day. No real data to back it up, though

    • Tracey 2.1

      I agree and think it is probably related tot he amoun to fmedia exposure a PM gets, compared to a LofO. Understandably the PM is a go to on many many things.

      I wonder, what would change, if anything, if published polls were outlawed in an election year?

    • Pasupial 2.2

      McFlock

      Polls do generally favour the incumbent. Most of the research on this is from a US setting, which I can’t be bothered churning through right now. This is the closest that I’ve been able to find to a NZ example thus far:

      http://www.digipoll.com/library/election-polling

      You can see the higher digipoll numbers than votes achieved on election night for the incumbent National in 2011 (51% vs 47%) and Labour in 2005 (45% vs 41%). The transition years contradict this however; in 1999 Labour was favoured in the poll (40% vs 39%), in 2008 National (48% vs 45%). Plus in 2002 National was favoured in the digipoll (23% vs 21%).

      I really need to get more NZ data on this though. The digipoll numbers seem to be averaged across the election year. Online; Roy Morgan only goes back to January 2012, and; Colmar Brunton to February 2011.

    • JK 2.3

      that didn’t happen with Helen Clark in her final term ….. media were very biased against her.

  3. Lanthanide 3

    “And the mechanism most people give for poll bias – the non-representative subset of the people who have landlines, should lead pollsters to overestimate New Zealand First with its older support base, not underestimate them. The underestimate of New Zealand First does not seem to fit the causal story people are telling.”

    Do elderly living in rest homes have their own individual phone lines? I’d suggest not all of them would.

    • Tracey 3.1

      If it is like other types of residential care there is a main line with a voice message allowing you to input the extension you require or else wait to get the reception?

      • Lanthanide 3.1.1

        Right, in which case there would only be 1 number in the phone book which would cover all residents, rather than 1 line per resident.

        So that would help to explain some of why NZ1st is under-represented.

  4. Ant 4

    How many people have their numbers listed these days anyway?

    • Pasupial 4.1

      Ant

      They don’t need a phone listing – they’ve got computers configured to generate random 7 digit numbers (plus area prefix). Or in Colmer Brunton speak; “Nationwide random digit dialling of landline telephones using stratified random probability sampling”.

      • Ant 4.1.1

        Cheers for that, for some reason I had it in my head that calling like that was legislated against and they had to use public information.

      • jaymam 4.1.2

        The random phone numbers are driven by a table for each telephone exchange that shows the telephone numbers actually in use. If those tables have not been updated since the program was supplied (by me!) to the pollster, the newly added numbers and new exchanges will not be polled. Those will tend to be in the outer cheaper suburbs where they are more likely to be lefties.

        • Pasupial 4.1.2.1

          Cheers for that info jaymam. Any idea how often those number tables are updated in the company you’ve had dealings with? [to remained unnamed obviously]

          • jaymam 4.1.2.1.1

            The director that I gave the program to ended up suing the programmer for a lot of money, for a different program. So I doubt that the tables have ever been updated. It is very possible they use a different program now, but they possibly think that nobody would notice if they don’t call all the new numbers.That director has now fallen out with the company that bears her name and now does some polling on her own, which has been widely criticised.

  5. fambo 5

    I wonder how difficult it would be for external agencies to manipulate polls to destabilise a government. They could, for instance, pump large sums of money into setting up their own polling company. Or ferret their operatives into positions of importance in extant polling companies.

  6. rain33 6

    Polls Shmolls, who can forget the embarrassing fail of the right wing, namely Dick Morris and Dean Chambers who had Romney winning the 2012 US election right up until the night of the election? Their methods of ‘unskewing’ the polls sounded quite justified, by re-weighting the sample to match what they believed the electorate would look like in terms of party identification, but as history has proven were completely off the mark.

    I would highly recommend the book The Signal and the Noise: Why Most Predictions Fail – but Some Don’t – by Nate Silver the ‘god of polling’ and American statistician, who in the 2012 US Presidential election correctly predicted the outcome of all 50 States and the District of Columbia, that same year his U.S. Senate race predictions were correct in 31 of 33 States, in 2008 he got 35/35

    Individual polls are an indication of a moment in time. However, when polls start to indicate a trend, that is worth paying attention to. That should be of a concern to Labour. The debate regarding cellphones versus landlines etc was discussed ad-nauseum in the US but became a mute point. For all we know people who have landlines are more likely to turn up and vote at the election booth than people who don’t have landlines. You will get giddy trying to work out all the variables. Instead Labour should be focusing on getting a cohesive message out to the voting public.

  7. Whatever next 7

    Do our polls use the same polling techniques as Nate Silver?

    • Lanthanide 7.1

      Nate Silver didn’t perform any polling himself, he just aggregated and number-crunched the polls that others did.

      In the process he uncovered statistical proof that one of the polling companies was deliberately biasing their polls towards the republicans – they shut up shop after that IIRC.

    • Tamati 7.2

      As said above, Silver doesn’t poll. Silver uses what’s called Bayesian statistics. It’s a totally different way of looking at sampling and prediction and is very trendy ATM.

      Unfortunately it would be very hard to replicate here because we too few polls and under MMP you don’t have an absolute ‘winner’ like you do in FPP,

  8. swordfish 8

    Here are the Poll averages from previous Election years for both (1) February (to help anticipate 2014 Election result from current (February) polling) and (2) the 4-week period immediately before the particular Election (with Election Result comparisons):

    2011

    Nat (Feb average) 53% / Election 47% (minus 6 points)
    Nat (Final 4-weeks average) 52% / Election 47% (minus 5 points)

    Lab (Feb average) 33% / Election 27% (minus 6 points)
    Lab (Final 4-weeks average) 28% / Election 27% (minus 1 point)

    Green (Feb average) 8% / Election 11% (plus 3 points)
    Green (Final 4-weeks average) 12% / Election 11% (minus 1 point)

    NZF (Feb average) 3% / Election 7% (plus 4 points)
    NZF (Final 4-weeks average) 3% / Election 7% (plus 4 points)

    2008

    Nat (Feb average) 53% / Election 45% (minus 8 points)
    Nat (Final 4-weeks average) 47% / Election 45% (minus 2 points)

    Lab (Feb average) 34% / Election 34% (Equal)
    Lab (Final 4-weeks average) 35% / Election 34% (minus 1 point)

    Green (Feb average) 6% / Election 7% (plus 1 point)
    Green (Final 4-week average) 8% / Election 7% (minus 1 point)

    NZF (Feb average) 3% / Election 4% (plus 1 point)
    NZF (Final 4-weeks average) 3% / Election 4% (plus 1 point)

    2005

    Nat (Feb average) 37% / Election 39% (plus 2 points)
    Nat (Final 4-weeks average) 40% / Election 39% (minus 1 point)

    Lab (Feb average) 45% / Election 41% (minus 4 points)
    Lab (Final 4-weeks average) 40% / Election 41% (plus 1 point)

    Green (Feb average) 5% / Election 5% (Equal)
    Green (Final 4-weeks average) 6% / Election 5% (minus 1 point)

    NZF (Feb average) 5% / Election 6% (plus 1 point)
    NZF (Final 4-weeks average) 6% / Election 6% (Equal)

    Need I say more ? (I need not).

  9. Crunchtime 9

    I was trying to find some info about opinion polls prior to the 2002 election but can’t seem to find any on the internet. There’s a lot of info on opinion polls for 2005 but not the previous…

    All I could find was this story about how polls were pointing to an outright majority, but people “withdrew their support” to deny them this. Clearly, National didn’t benefit from this either, nearly slipping below 20% support, which I believe is far lower than Labour’s support has ever fallen.

    I also found some interesting food for thought: the “Electoral (Public Opinion Polls) Amendment Bill” 2002. This is to outlaw any publicity whatsoever 28 days prior to the election. Because OPINION POLLS INFLUENCE RESULTS. Interesting huh… As if banning them for 28 days is going to make any difference!

    Election results 2002:

    Labour 41.3%

    National 20.9%

    NZ First 10.4%

    Act 7.1%

    Green 7%

    United Future 6.7%

    Progressive 1.7%

    Possibly the most even spread of votes in NZ election history?

    It would be interesting to see what opinion polls were doing leading up to this result.

Recent Comments

Recent Posts

  • Environment Court Judge appointed
    Prudence Steven QC, barrister of Christchurch has been appointed as an Environment Judge and District Court Judge to serve in Christchurch, Attorney-General David Parker announced today. Ms Steven has been a barrister sole since 2008, practising in resource management and local government / public law.    She was appointed a Queen’s ...
    BeehiveBy beehive.govt.nz
    8 hours ago
  • Government moves on climate promises
    The Government is delivering on its first tranche of election promises to take action on climate change with a raft of measures that will help meet New Zealand’s 2050 carbon neutral target, create new jobs and boost innovation. “This will be an ongoing area of action but we are moving ...
    BeehiveBy beehive.govt.nz
    11 hours ago
  • Jump starting research careers
    The Government is investing up to $10 million to support 30 of the country’s top early-career researchers to develop their research skills. “The pandemic has had widespread impacts across the science system, including the research workforce. After completing their PhD, researchers often travel overseas to gain experience but in the ...
    BeehiveBy beehive.govt.nz
    12 hours ago
  • Project protects jobs and nature
    A Waitomo-based Jobs for Nature project will keep up to ten people employed in the village as the tourism sector recovers post Covid-19 Conservation Minister Kiri Allan says. “This $500,000 project will save ten local jobs by deploying workers from Discover Waitomo into nature-based jobs. They will be undertaking local ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Minister Shaw speaks with U.S. Presidential Envoy John Kerry
    Minister for Climate Change, James Shaw spoke yesterday with President Biden’s Special Presidential Envoy for Climate John Kerry. “I was delighted to have the opportunity to speak with Mr. Kerry this morning about the urgency with which our governments must confront the climate emergency. I am grateful to him and ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Minister of Foreign Affairs makes three diplomatic appointments
    Foreign Affairs Minister Hon Nanaia Mahuta today announced three diplomatic appointments: Alana Hudson as Ambassador to Poland John Riley as Consul-General to Hong Kong Stephen Wong as Consul-General to Shanghai   Poland “New Zealand’s relationship with Poland is built on enduring personal, economic and historical connections. Poland is also an important ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Major redevelopment of Wainuiomata High School underway
    Work begins today at Wainuiomata High School to ensure buildings and teaching spaces are fit for purpose, Education Minister Chris Hipkins says. The Minister joined principal Janette Melrose and board chair Lynda Koia to kick off demolition for the project, which is worth close to $40 million, as the site ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • New expert group appointed to advise Government on Oranga Tamariki
    A skilled and experienced group of people have been named as the newly established Oranga Tamariki Ministerial Advisory Board by Children’s Minister Kelvin Davis today. The Board will provide independent advice and assurance to the Minister for Children across three key areas of Oranga Tamariki: relationships with families, whānau, and ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • COVID-19 vaccine slated for possible approval next week
    The green light for New Zealand’s first COVID-19 vaccine could be granted in just over a week, Prime Minister Jacinda Ardern said today. “We’re making swift progress towards vaccinating New Zealanders against the virus, but we’re also absolutely committed to ensuring the vaccines are safe and effective,” Jacinda Ardern said. ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • New ACC Board members announced.
    The Minister for ACC is pleased to announce the appointment of three new members to join the Board of ACC on 1 February 2021. “All three bring diverse skills and experience to provide strong governance oversight to lead the direction of ACC” said Hon Carmel Sepuloni. Bella Takiari-Brame from Hamilton ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Economic boost for Southland marae
    The Government is investing $9 million to upgrade a significant community facility in Invercargill, creating economic stimulus and jobs, Infrastructure Minister Grant Robertson and Te Tai Tonga MP Rino Tirikatene have announced.  The grant for Waihōpai Rūnaka Inc to make improvements to Murihiku Marae comes from the $3 billion set ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Celebrating the Entry Into Force of the Treaty on the Prohibition of Nuclear Weapons
    [Opening comments, welcome and thank you to Auckland University etc] It is a great pleasure to be here this afternoon to celebrate such an historic occasion - the entry into force of the Treaty on the Prohibition of Nuclear Weapons. This is a moment many feared would never come, but ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Supporting disabled people to stay connected
    The Government is providing $3 million in one-off seed funding to help disabled people around New Zealand stay connected and access support in their communities, Minister for Disability Issues, Carmel Sepuloni announced today. The funding will allow disability service providers to develop digital and community-based solutions over the next two ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Voluntary saliva testing offered to quarantine workers from Monday
    Border workers in quarantine facilities will be offered voluntary daily COVID-19 saliva tests in addition to their regular weekly testing, COVID-19 Response Minister Chris Hipkins said today. This additional option will be rolled out at the Jet Park Quarantine facility in Auckland starting on Monday 25 January, and then to ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Next steps in firearms buy-back
    The next steps in the Government’s ambitious firearms reform programme to include a three-month buy-back have been announced by Police Minister Poto Williams today.  “The last buy-back and amnesty was unprecedented for New Zealand and was successful in collecting 60,297 firearms, modifying a further 5,630 firearms, and collecting 299,837 prohibited ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Jobs for Nature projects target iconic ecosystems
    Upscaling work already underway to restore two iconic ecosystems will deliver jobs and a lasting legacy, Conservation Minister Kiri Allan says.  “The Jobs for Nature programme provides $1.25 billion over four years to offer employment opportunities for people whose livelihoods have been impacted by the COVID-19 recession. “Two new projects ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • New Public Housing Plan announced
    The Government has released its Public Housing Plan 2021-2024 which outlines the intention of where 8,000 additional public and transitional housing places announced in Budget 2020, will go. “The Government is committed to continuing its public house build programme at pace and scale. The extra 8,000 homes – 6000 public ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Prime Minister congratulates President Joe Biden on his inauguration
    Prime Minister Jacinda Ardern has congratulated President Joe Biden on his inauguration as the 46th President of the United States of America. “I look forward to building a close relationship with President Biden and working with him on issues that matter to both our countries,” Jacinda Ardern said. “New Zealand ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Jobs for Nature funding will create training and employment opportunities
    A major investment to tackle wilding pines in Mt Richmond will create jobs and help protect the area’s unique ecosystems, Biosecurity Minister Damien O’Connor says. The Mt Richmond Forest Park has unique ecosystems developed on mineral-rich geology, including taonga plant species found nowhere else in the country. “These special plant ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Pre-departure testing extended to all passengers to New Zealand
    To further protect New Zealand from COVID-19, the Government is extending pre-departure testing to all passengers to New Zealand except from Australia, Antarctica and most Pacific Islands, COVID-19 Response Minister Chris Hipkins said today. “The change will come into force for all flights arriving in New Zealand after 11:59pm (NZT) on Monday ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Bay Cadets learn skills to protect environment
    Bay Conservation Cadets launched with first intake Supported with $3.5 million grant Part of $1.245b Jobs for Nature programme to accelerate recover from Covid Cadets will learn skills to protect and enhance environment Environment Minister David Parker today welcomed the first intake of cadets at the launch of the Bay ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Cook Islanders to resume travel to New Zealand
    The Prime Minister of New Zealand Jacinda Ardern and the Prime Minister of the Cook Islands Mark Brown have announced passengers from the Cook Islands can resume quarantine-free travel into New Zealand from 21 January, enabling access to essential services such as health. “Following confirmation of the Cook Islands’ COVID ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Supporting communities and landowners to grow employment opportunities
    Jobs for Nature funding is being made available to conservation groups and landowners to employ staff and contractors in a move aimed at boosting local biodiversity-focused projects, Conservation Minister Kiritapu Allan has announced. It is estimated some 400-plus jobs will be created with employment opportunities in ecology, restoration, trapping, ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Border exception for some returning international tertiary students
    The Government has approved an exception class for 1000 international tertiary students, degree level and above, who began their study in New Zealand but were caught offshore when border restrictions began. The exception will allow students to return to New Zealand in stages from April 2021. “Our top priority continues ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Tiwai deal gives time for managed transition
    Today’s deal between Meridian and Rio Tinto for the Tiwai smelter to remain open another four years provides time for a managed transition for Southland. “The deal provides welcome certainty to the Southland community by protecting jobs and incomes as the region plans for the future. The Government is committed ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • New member for APEC Business Advisory Council
    Prime Minister Jacinda Ardern has appointed Anna Curzon to the APEC Business Advisory Council (ABAC). The leader of each APEC economy appoints three private sector representatives to ABAC. ABAC provides advice to leaders annually on business priorities. “ABAC helps ensure that APEC’s work programme is informed by business community perspectives ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Govt’s careful economic management recognised
    The Government’s prudent fiscal management and strong policy programme in the face of the COVID-19 global pandemic have been acknowledged by the credit rating agency Fitch. Fitch has today affirmed New Zealand’s local currency rating at AA+ with a stable outlook and foreign currency rating at AA with a positive ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Additional actions to keep COVID-19 out of NZ
    The Government is putting in place a suite of additional actions to protect New Zealand from COVID-19, including new emerging variants, COVID-19 Response Minister Chris Hipkins said today. “Given the high rates of infection in many countries and evidence of the global spread of more transmissible variants, it’s clear that ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • 19 projects will clean up and protect waterways
    $36 million of Government funding alongside councils and others for 19 projects Investment will clean up and protect waterways and create local jobs Boots on the ground expected in Q2 of 2021 Funding part of the Jobs for Nature policy package A package of 19 projects will help clean up ...
    BeehiveBy beehive.govt.nz
    3 weeks ago