web analytics
The Standard

Polity: X more thoughts on poll bias

Written By: - Date published: 11:36 am, February 26th, 2014 - 26 comments
Categories: national, nz first, polls - Tags: , ,

Rob Salmond at Polity has some changing thoughts on the subject of poll bias.

Whenever the media report on polls, they get accused of all manner of biases by the blinkered commentariat. Most of it is nonsense, of course. But some criticism of the polling methods themselves may have more merit that I first thought.

I have been keeping track of the polls in New Zealand for a few years now, mainly trying to get the rightindustry average level of support for each party, and figuring out ways to account for the biases in individual pollsters’ methods. Polity’s Poll of Polls is part of this work, as was the academic work I did uncovering bias in the old version of the Colmar Brunton poll.

But both Danyl at the Dim-Post and Gavin White at UMR have been looking at something more fundamental – is the polling industry biased as a whole? There is certainly a small enough number of competing firms in the market, and enough limitations on their work (e.g. the cost of calling mobiles) for bias to conceivably sneak in. For example, the best information I have seen suggests that the New Zealand population with access to a landline skew about 1.7% more towards National than the population as a whole. Calling a random selection of landlines, even if you do some weighting by demographics, can easily give you a result reflecting this bias.

Is this what is happening in New Zealand? Using quite different methods (but the same raw data), Danyl and Gavin both come to the conclusion that the New Zealand polling industry overstates National, modestly overstates the Greens, understates New Zealand First, and gets Labour about right.

I can see reasons other than poll bias why election results for the Greens may not reflect their polling. It could easily be that the young people who make up a lot of the Greens’ support base are just not very reliable when it comes to turning out in elections.

And the mechanism most people give for poll bias – the non-representative subset of the people who have landlines, should lead pollsters to overestimate New Zealand First with its older support base, not underestimate them. The underestimate of New Zealand First does not seem to fit the causal story people are telling.

But the fairly consistent results on National are harder to dismiss. In each of the last three elections we have been left staring at the chicken bones trying to explain how National’s support seemed to fall away in the dying days. A delayed Brethren effect? Collective panic at the thought of one party government? Maybe. Or maybe these are just the stories we tell ourselves in order to feel comfortable with the polls.

(Having said that, the polls in 2008 and 2011 did actually show a late-term decline for National, just not one big enough to account for the actual result. So the pollsters were not at all surprised in either case that National’s support was lower than its early campaign peak, but they were surprised just how far National had fallen.)

I was caught short in 2011 relying on the polling industry average by falsely projecting that National would get enough votes to govern alone, and that New Zealand First would not break 5%. I was wrong on both counts, and in exactly the direction that Danyl and Gavin would predict. Some might say that is all the evidence you need of bias.

Ultimately, however, I am not sold just yet on the notion of industry-level bias.

The studies Danyl and Gavin have done here can claim as many poll-level observations as they like, but in an important sense the number of elections is the number of observations, because that is the number of times you observe the true value of the things we are busy estimating. Which means right now we are operating on N equals three. Making conclusions on an N this small is risky.

While I’m not yelling “fire” at this stage, I do think there may be smoke. I did not think that three years ago.

If later this year we have to wave our hands at yet another inexplicable final week decline for National, and again marvel at Winston’s ability to sneak up on everyone at the last second, then that will substantially strengthen the claim of industry-wide bias.

If we find there is this bias down the track, then we will have to look very carefully at the way in which a slanted perception of the political reality today impacts on voters’ minds, usually through the journalistic filter. Do they unfairly nurture the reputations of over-estimated parties like National and the Greens, and unfairly tarnish New Zealand First. Does it have real consequences for those parties at the ballot box in an MMP environment? And, of so, is that political speech that should be protected, or protected against?

26 comments on “Polity: X more thoughts on poll bias”

  1. Blue 1

    Do they unfairly nurture the reputations of over-estimated parties like National and the Greens, and unfairly tarnish New Zealand First.

    The media don’t really care what the poll results are when it comes to how they represent certain parties. Witness two pieces in the Herald today focused on the 0% wonder.

    Their beef with NZ First is mostly journos having personal feuds with Winston Peters.

    If anything, the poll results which overestimate National simply give the media more of an excuse to downplay Labour, which they do anyway.

    • Crunchtime 1.1

      That doesn’t necessarily hold true when you are talking about a party with a large support base. If it looks (falsely) like a foregone conclusion that a party will win a majority vote, or close to, then it’s likely to demoralise and demotivate those who might vote against them.

      I’m not saying this is entirely to blame for the record-breaking 800,000 who didn’t vote last election, but I would suggest it’s one of many contributing factors.

    • Crunchtime 1.2

      …in addition, isn’t there the type of person who tends to vote with who seems to be the “winner” in order to give a “stable government”?

      If so then polls that show National as reasonably far ahead will have a gravitational effect, as it were, on future poll results favouring National, including the ultimate poll results of the election.

      Furthermore isn’t this a big reason why Labour has been polling so poorly for the past half decade, because it has been well publicised as being “divided” and therefore “unstable”? To be fair, Labour haven’t really done enough to dispel that notion.

  2. McFlock 2

    It also seems to me that there seems to be a bit of a bias towards the government of the day. No real data to back it up, though

    • Tracey 2.1

      I agree and think it is probably related tot he amoun to fmedia exposure a PM gets, compared to a LofO. Understandably the PM is a go to on many many things.

      I wonder, what would change, if anything, if published polls were outlawed in an election year?

    • Pasupial 2.2

      McFlock

      Polls do generally favour the incumbent. Most of the research on this is from a US setting, which I can’t be bothered churning through right now. This is the closest that I’ve been able to find to a NZ example thus far:

      http://www.digipoll.com/library/election-polling

      You can see the higher digipoll numbers than votes achieved on election night for the incumbent National in 2011 (51% vs 47%) and Labour in 2005 (45% vs 41%). The transition years contradict this however; in 1999 Labour was favoured in the poll (40% vs 39%), in 2008 National (48% vs 45%). Plus in 2002 National was favoured in the digipoll (23% vs 21%).

      I really need to get more NZ data on this though. The digipoll numbers seem to be averaged across the election year. Online; Roy Morgan only goes back to January 2012, and; Colmar Brunton to February 2011.

    • JK 2.3

      that didn’t happen with Helen Clark in her final term ….. media were very biased against her.

  3. Lanthanide 3

    “And the mechanism most people give for poll bias – the non-representative subset of the people who have landlines, should lead pollsters to overestimate New Zealand First with its older support base, not underestimate them. The underestimate of New Zealand First does not seem to fit the causal story people are telling.”

    Do elderly living in rest homes have their own individual phone lines? I’d suggest not all of them would.

    • Tracey 3.1

      If it is like other types of residential care there is a main line with a voice message allowing you to input the extension you require or else wait to get the reception?

      • Lanthanide 3.1.1

        Right, in which case there would only be 1 number in the phone book which would cover all residents, rather than 1 line per resident.

        So that would help to explain some of why NZ1st is under-represented.

  4. Ant 4

    How many people have their numbers listed these days anyway?

    • Pasupial 4.1

      Ant

      They don’t need a phone listing – they’ve got computers configured to generate random 7 digit numbers (plus area prefix). Or in Colmer Brunton speak; “Nationwide random digit dialling of landline telephones using stratified random probability sampling”.

      • Ant 4.1.1

        Cheers for that, for some reason I had it in my head that calling like that was legislated against and they had to use public information.

      • jaymam 4.1.2

        The random phone numbers are driven by a table for each telephone exchange that shows the telephone numbers actually in use. If those tables have not been updated since the program was supplied (by me!) to the pollster, the newly added numbers and new exchanges will not be polled. Those will tend to be in the outer cheaper suburbs where they are more likely to be lefties.

        • Pasupial 4.1.2.1

          Cheers for that info jaymam. Any idea how often those number tables are updated in the company you’ve had dealings with? [to remained unnamed obviously]

          • jaymam 4.1.2.1.1

            The director that I gave the program to ended up suing the programmer for a lot of money, for a different program. So I doubt that the tables have ever been updated. It is very possible they use a different program now, but they possibly think that nobody would notice if they don’t call all the new numbers.That director has now fallen out with the company that bears her name and now does some polling on her own, which has been widely criticised.

  5. fambo 5

    I wonder how difficult it would be for external agencies to manipulate polls to destabilise a government. They could, for instance, pump large sums of money into setting up their own polling company. Or ferret their operatives into positions of importance in extant polling companies.

  6. rain33 6

    Polls Shmolls, who can forget the embarrassing fail of the right wing, namely Dick Morris and Dean Chambers who had Romney winning the 2012 US election right up until the night of the election? Their methods of ‘unskewing’ the polls sounded quite justified, by re-weighting the sample to match what they believed the electorate would look like in terms of party identification, but as history has proven were completely off the mark.

    I would highly recommend the book The Signal and the Noise: Why Most Predictions Fail – but Some Don’t – by Nate Silver the ‘god of polling’ and American statistician, who in the 2012 US Presidential election correctly predicted the outcome of all 50 States and the District of Columbia, that same year his U.S. Senate race predictions were correct in 31 of 33 States, in 2008 he got 35/35

    Individual polls are an indication of a moment in time. However, when polls start to indicate a trend, that is worth paying attention to. That should be of a concern to Labour. The debate regarding cellphones versus landlines etc was discussed ad-nauseum in the US but became a mute point. For all we know people who have landlines are more likely to turn up and vote at the election booth than people who don’t have landlines. You will get giddy trying to work out all the variables. Instead Labour should be focusing on getting a cohesive message out to the voting public.

  7. Whatever next 7

    Do our polls use the same polling techniques as Nate Silver?

    • Lanthanide 7.1

      Nate Silver didn’t perform any polling himself, he just aggregated and number-crunched the polls that others did.

      In the process he uncovered statistical proof that one of the polling companies was deliberately biasing their polls towards the republicans – they shut up shop after that IIRC.

    • Tamati 7.2

      As said above, Silver doesn’t poll. Silver uses what’s called Bayesian statistics. It’s a totally different way of looking at sampling and prediction and is very trendy ATM.

      Unfortunately it would be very hard to replicate here because we too few polls and under MMP you don’t have an absolute ‘winner’ like you do in FPP,

  8. swordfish 8

    Here are the Poll averages from previous Election years for both (1) February (to help anticipate 2014 Election result from current (February) polling) and (2) the 4-week period immediately before the particular Election (with Election Result comparisons):

    2011

    Nat (Feb average) 53% / Election 47% (minus 6 points)
    Nat (Final 4-weeks average) 52% / Election 47% (minus 5 points)

    Lab (Feb average) 33% / Election 27% (minus 6 points)
    Lab (Final 4-weeks average) 28% / Election 27% (minus 1 point)

    Green (Feb average) 8% / Election 11% (plus 3 points)
    Green (Final 4-weeks average) 12% / Election 11% (minus 1 point)

    NZF (Feb average) 3% / Election 7% (plus 4 points)
    NZF (Final 4-weeks average) 3% / Election 7% (plus 4 points)

    2008

    Nat (Feb average) 53% / Election 45% (minus 8 points)
    Nat (Final 4-weeks average) 47% / Election 45% (minus 2 points)

    Lab (Feb average) 34% / Election 34% (Equal)
    Lab (Final 4-weeks average) 35% / Election 34% (minus 1 point)

    Green (Feb average) 6% / Election 7% (plus 1 point)
    Green (Final 4-week average) 8% / Election 7% (minus 1 point)

    NZF (Feb average) 3% / Election 4% (plus 1 point)
    NZF (Final 4-weeks average) 3% / Election 4% (plus 1 point)

    2005

    Nat (Feb average) 37% / Election 39% (plus 2 points)
    Nat (Final 4-weeks average) 40% / Election 39% (minus 1 point)

    Lab (Feb average) 45% / Election 41% (minus 4 points)
    Lab (Final 4-weeks average) 40% / Election 41% (plus 1 point)

    Green (Feb average) 5% / Election 5% (Equal)
    Green (Final 4-weeks average) 6% / Election 5% (minus 1 point)

    NZF (Feb average) 5% / Election 6% (plus 1 point)
    NZF (Final 4-weeks average) 6% / Election 6% (Equal)

    Need I say more ? (I need not).

  9. Crunchtime 9

    I was trying to find some info about opinion polls prior to the 2002 election but can’t seem to find any on the internet. There’s a lot of info on opinion polls for 2005 but not the previous…

    All I could find was this story about how polls were pointing to an outright majority, but people “withdrew their support” to deny them this. Clearly, National didn’t benefit from this either, nearly slipping below 20% support, which I believe is far lower than Labour’s support has ever fallen.

    I also found some interesting food for thought: the “Electoral (Public Opinion Polls) Amendment Bill” 2002. This is to outlaw any publicity whatsoever 28 days prior to the election. Because OPINION POLLS INFLUENCE RESULTS. Interesting huh… As if banning them for 28 days is going to make any difference!

    Election results 2002:

    Labour 41.3%

    National 20.9%

    NZ First 10.4%

    Act 7.1%

    Green 7%

    United Future 6.7%

    Progressive 1.7%

    Possibly the most even spread of votes in NZ election history?

    It would be interesting to see what opinion polls were doing leading up to this result.

Recent Comments

Recent Posts

1 2 3 8

  • Tragedy must be impetus for better training
    The Police Minister needs to explain why unsworn and inadequately trained custody officers were put in a situation of caring for a medically unwell prisoner on a busy Saturday night, Labour’s Police spokesperson Kelvin Davis says. Commenting on an IPCA… ...
    2 days ago
  • Government must be more transparent on investor state clauses
    The Government must be more transparent around the draft investor state dispute settlements in the TPPA, says David Parker, Labour’s Export Growth and Trade spokesperson. “Labour is pro trade, and is proud of the FTA we negotiated with China, which… ...
    2 days ago
  • Protect university staff and student voices
    The Green Party believes ensuring student and staff representation on university councils is important. National recently passed a law reducing the size of university governance councils while increasing the proportion of the members nominated by, guess who… Steven Joyce. The… ...
    GreensBy Gareth Hughes MP
    3 days ago
  • C’mon Nick what’s the truth on the RMA
     “Nick Smith has got to fess up and tell us what is happening to his much vaunted RMA reform, Labour’s Environment spokesperson Megan Woods says.  “With just a day and a half to go before the polls open in Northland,… ...
    3 days ago
  • SSC salaries sink National’s spending spin
    Massive pay rises at the State Services Commission prove National’s claims of clamping down on spending in the public sector are simply fantasy, Labour’s State Services spokesman Kris Faafoi says. “Salaries in this one department are almost $70,000 more than… ...
    3 days ago
  • We can fix Christchurch and keep our assets
    The Christchurch City Council is seeking public feedback on its proposed 10 year plan for Council revenue and spending. This is probably one of the most significant 10 year plans ever to be written by a local council because of… ...
    GreensBy Eugenie Sage MP
    3 days ago
  • Epidemic of serious assaults in our prisons
    Labour wants stab proof vests and pepper spray for all corrections officers to keep them safe from the epidemic of serious prison assaults that are occurring around the country’s jails, says Labour’s Corrections Spokesperson Kelvin Davis.  “There have been five… ...
    3 days ago
  • Listen to the locals Hekia!
    Minister Hekia Parata needs to understand what consultation is, Port Hills MP Ruth Dyson says. “It means you have to listen to what people say in their submissions and then be able to demonstrate you have considered their views when… ...
    4 days ago
  • Thanking our caregivers
    Let’s celebrate and thank our caregivers. This week is caregivers’ week. It’s a chance to acknowledge the thousands of women, and occasional other person, who are caring for the elderly and disabled in our country. They hold people’s lives in… ...
    GreensBy Jan Logie MP
    4 days ago
  • Mana Post shop the best outcome for community
    Labour MP for Mana Kris Faafoi has welcomed the move to place the services from the Mana Post shop to a local small business. “This is the best outcome for the community we could ask for. All the vital services… ...
    4 days ago
  • Labour looks to put the tea back into entitlements
    Labour is moving to restore the rights of Kiwis to take tea and rest breaks, Leader Andrew Little says. “Within months of the Government’s Employment Relations Amendment Bill becoming law we are already seeing some of our largest companies, including… ...
    4 days ago
  • Desperate money grab to keep Ruataniwha afloat
    The Hawke’s Bay Regional Investment Company’s decision to borrow $4 million to keep the Ruataniwha project afloat is a case of throwing ratepayer’s good money after bad, says Labour’s Water spokesperson Meka Whaitiri and Napier MP Stuart Nash.   “This bridging… ...
    5 days ago
  • Roundup: UN finds it “probably” causes cancer
    At last the UN has spoken out against the widely-used weedkiller Roundup. The UN’s International Agency for Research on Cancer has identified glyphosate, the principle ingredient in Roundup, as a probable carcinogen. They also include as probable carcinogens the insecticides… ...
    GreensBy Steffan Browning MP
    5 days ago
  • Invermay petition delivered to Parliament
    Labour’s Economic Development spokesperson David Clark handed over a 12,450 signature Save Invermay petition to Dunedin South MP Clare Curran on the steps of Parliament today.  “The level of support that the petition has received across New Zealand is overwhelming,”… ...
    5 days ago
  • Redcliffs School closure plan wrong
    The Government’s proposal to consult on the closure of Redcliffs School not only goes against the best geotechnical advice, but more importantly goes against the best educational outcomes for Redcliffs children and the health of our community, Port Hills MP… ...
    5 days ago
  • Cotton On first to test the tea breaks law
    Australian corporate Cotton On, the first major business operating in New Zealand to exploit the new tea breaks law, could walk away from negotiations if it doesn’t get its own way, says Labour Party Labour Relations spokesperson Iain Lees-Galloway.  “Cotton… ...
    6 days ago
  • World water day: eight rivers in one day
    Our photo journey started by the Waioweka (also known as Waioeka) River which flows from Te Urewera to Opotiki, and is surrounded by beautiful forest. The water looked great! Kopeopeo Canal It contrasted greatly with the Kopeopeo Canal near Whakatane,… ...
    GreensBy Catherine Delahunty MP
    6 days ago
  • Council can stop Port’s encroachment on harbour
    As owner of the Port of Auckland, Council can stop the wharf extension and reclamation if it wants to, says Labour’s Auckland Issues spokesperson Phil Goff. ‘As owner the council is custodian of the port and harbour on behalf of… ...
    6 days ago
  • We all benefit when education meets everyone’s needs
    As Dyslexia week comes to a close,  Dyslexia NZ have reminded us that around 10% of our citizens are dyslexic and are entitled to better support. One of their strongest arguments is that failure to provide identification and support for… ...
    GreensBy Catherine Delahunty MP
    6 days ago
  • State house sell-off fiasco a gift for developers
    The Government’s property developer mates are the only people who can salvage National’s state house sell-off now the Salvation Army has torpedoed the policy, Labour’s Housing spokesperson Phil Twyford says. “Having been cynically used by the Government as the poster… ...
    6 days ago
  • National reinforces inequality in schools
    The National Government’s flagship programme Investing in Educational Success is clearly reinforcing inequality in the school system, Labour’s Education spokesperson Chris Hipkins says. “The analysis released today by the NZEI clearly shows schools in wealthier suburbs are the main beneficiaries… ...
    6 days ago
  • Big change starts small
    Today marks Race Relations Day in New Zealand. Race Relations Day coincides with the International Day for the Elimination of Racial Discrimination.  The United Nations General Assembly chose this day as it marks the day in 1960 when 69 peaceful… ...
    GreensBy Denise Roche MP
    1 week ago
  • Israel, Palestine and the question of statehood
    The knife-edge election in Israel complicates the Middle East situation, even more than usual. The Prime Minister-elect, Binyamin Netanyahu, is moving to form a government. Netanyahu has indicated that, during his term, a Palestinian state would not be established. That… ...
    GreensBy Kennedy Graham MP
    1 week ago
  • Christchurch transport goes backwards
    The Green Party has a vision of a liveable, accessible Christchurch with a sense of identity and strong connected communities. Instead, 2013 census figures released by Statistics New Zealand reveal a fractured community, and tell a story of frustrated Christchurch commuters… ...
    GreensBy Eugenie Sage MP
    1 week ago
  • Super Fund should divest $140 million in high risk coal
    The Green Party is calling on the New Zealand Super Fund to divest their $140 million investment in coal companies that are vulnerable to becoming financially stranded according to a damning new report from Oxford University. The Smith School of… ...
    GreensBy Russel Norman MP
    1 week ago
  • Learn to count with Mark Osborne: 0 + 1 = ?
    The adage about the first casualty of war being truth is one that might often be applied to the political battle for hearts and minds, and of course votes. A rather unfortunate example of this has been arriving in the… ...
    GreensBy David Clendon MP
    2 weeks ago
  • Is it still a safety net when the holes are this big?
    Over the last few weeks I’ve been wondering how safe our income support system is for people, especially those with cognitive or learning disabilities. I’ve been trying to support a young man who was severely injured in a workplace accident… ...
    GreensBy Jan Logie MP
    2 weeks ago
  • Pasifika – protecting the Pacific needed now more than ever.
    Over the weekend thousands of Aucklanders flocked to celebrate our city’s diverse Pacific communities and cultures at the annual Pasifika festival and the Greens were there to join them. The Pasifika festival has been held every year for 23… ...
    GreensBy Denise Roche MP
    2 weeks ago
  • Sounds Stakeholders Seek a Sustainable Future
    It was heartening to see a large number of people who care about the Marlborough Sounds come together at the Marlborough Marine Futures’ forum in Picton on March 8. Fellow Green MP Steffan Browning, who lives in Marlborough, and I… ...
    GreensBy Eugenie Sage MP
    2 weeks ago
  • Solid Energy, who will clean up the mess?
    What can you say? This state-owned coal miner is facing some very serious problems. They haven’t run a profit in years, have required two Government bailouts, laid-off more than 700 staff and look like they need a third injection of… ...
    GreensBy Gareth Hughes MP
    2 weeks ago

Removed at the request of The Daily Blog.
Public service advertisements by The Standard

Current CO2 level in the atmosphere