Polls

Written By: - Date published: 8:22 am, September 28th, 2016 - 61 comments
Categories: making shit up, Media, polls - Tags:

While politicos were concentrating on Hills defeating The Donald in the first US debate*, Roy Morgan released their latest poll, showing Labour massively up and National down a chunk.

It brings Roy Morgan into line with the latest Newshub poll, and Labour’s UMR poll.  Labour low 30s, National low 40s, Labour-Greens at or just above National.

The likes of Matthew Hooton are obviously rubbishing it:

https://twitter.com/MatthewHootonNZ/status/780673622515916800

But will we see the equivalent articles about how there must be panic (this time in National), like the last lot about Labour by a gender-balancing obssessed media?  (a few bloke MPs lose their jobs… and?)

Roy Morgan is generally not covered well in the media as it’s not connected to a news organisation and because it has the reputation of “it moves around a lot.”  And it does.  But part of that is because it actually has enough data points to see that: all polls move around quite a bit.  When you only get polls every 2 months (OneNews), or 3 times a year (Newshub) or indeed haven’t been seen since 2015 (Herald), suddenly every point on your graph seems highly significant.

Odds are that One News’ last one that set off such a media flurry was an outlier (95% confidence of +/- 3.6%?  Or 1 in 20 polls is just plain wrong?).  But we have so few data points we can’t tell.

So really it’s a great shame that Roy Morgan have moved from fortnightly to monthly.  And that apparently our media can’t afford the number of polls to actually tell us much.  And when they do spend the money they feel so compelled to make ‘good’ use of it, they massively over-hype what it’s actually telling us.

If you’re looking at fivethirtyeight (I know you are…) and the poll updates, you see massive discrepancies between polls.  Between different polls that Nate Silver is quite happy with their methodology, you’ll see Clinton nationally up by 6 and down by 2 on the same day.  If you look at the likes of Utah, Clinton could be up by 4 or down by 39 (okay the up by 4 is GCS, but there’s still >30 points between other surveys).   And the nationwide tracking polls have big swings across a few days despite having consistent methodology and not always an event to attribute any swing too.

You end up looking at trends and averages, and you need enough data points to do that.  We don’t have them.  The Radio NZ poll of polls is better than any individual poll, but they’re still ultimately stuck with too little data.

So: I propose our media scrap polls until they have enough money to do them properly.  Otherwise they paint a narrative that can become self-fulfilling on too little data – or they end up looking like dolts when they have to suddenly declare black is white when their next result comes out a few months later…

–

* we’ll have to wait a few days for the polls to validate my statement… 🙂

[And Gary Morgan: just stick to describing the results, when you try to give a reason for the latest vast swing you look silly.  When National are up 6 points because the news has all been about homelessness and the housing crisis… it’s just best not to try and make up why]

61 comments on “Polls ”

  • Comments are now closed

  1. Lanthanide 1
    28 September 2016 at 9:45 am

    “Or 1 in 20 polls is just plain wrong?”

    This needs to be repeated every time a poll is released. Of course the media never do it. I suspect that the news reporters, or perhaps anyone at the media companies, don’t actually understand what the margin of error means. They just dutifully report it because “every poll has a margin of error” as if that gives them cover for all the ridiculous commentary they’ve just made.

    But your premise that we don’t have enough polls, so stop doing them entirely, is silly. I get the point you’re trying to make – that bad data can be misleading, but even with Brexit, the point you could take away from that is that it was a very close race – in the end leave won and it was a ‘surprise’, but it probably shouldn’t have been.

    • Colonial Viper 1.1
      28 September 2016 at 9:52 am

      You mean that 19 out of 20 Roy Morgans/Colmar Bruntons/UMRs/etc are accurate?

      That’s got to be some kind of optimism.

      • Lanthanide 1.1.1
        28 September 2016 at 10:04 am

        “You mean that 19 out of 20 Roy Morgans/Colmar Bruntons/UMRs/etc are accurate?

        That’s got to be some kind of optimism.”

        No, I mean that assuming all of the methodology is sound, 1 out of 20 polls will be wrong by greater than the margin of error.

        This is based on the assumption that the methodology is sound, and frankly I think that’s probably not a safe assumption for NZ pollsters.

        • Draco T Bastard 1.1.1.1
          28 September 2016 at 12:18 pm

          This is based on the assumption that the methodology is sound, and frankly I think that’s probably not a safe assumption for NZ pollsters.

          QFT

          • billmurray 1.1.1.1.1
            28 September 2016 at 1:00 pm

            Draco T Bastard, agree with you about the soundness or otherwise of NZ polling companies.
            Nontheless it seems that congrats are in order for Labour,
            When is the next poll due ?????????????.

        • Matthew 1.1.1.2
          28 September 2016 at 2:50 pm

          Good thing Roy Morgan are Australian pollsters, then. ^_^

        • Nic the NZer 1.1.1.3
          28 September 2016 at 8:04 pm

          Oh no its far worse than that in fact. The 19 out of 20 is just saying that if the same poll is conducted over the same period then 19 of 20 times it is expected to give a similar result.

          But being polled (not a euphamism) is not the same activity as voting. For example your answer to a poll effects the poll result, but your vote may decide an election some people may answer differently to poll questions. Or refuse to participate when they intend to vote, why would people who don’t answer polls be an unbiased subsample of the total anyway? Or with a poll the pollster calls you with a vote you go to the ballot box. So is calling people an unbiased reflection of this fact?

          No sound methodology can really deal with any of these realities of polling.

    • RedLogix 1.2
      28 September 2016 at 9:54 am

      Yes … polls are inherently statistical beasts. I make no special claim to be a skilled statistician, but I know just enough to always be cautious of reading too much into small data sets.

  2. RedLogix 2
    28 September 2016 at 9:49 am

    Still too soon to break out the chardonnay. Eyeballing the trend lines in the OP it’s clear that National are still just within their historic ‘expected range’. I agree the direction is down, but they’ve been this low before and bounced back.

    I want to see them consistently below 40% for a couple of polls before we can start talking about a real change of sentiment in the electorate causing a sustained collapse of their core support. It isn’t clear that’s happened yet.

    Certainly from my social networking there is a lot of disquiet about the direction NZ is heading. Most people aren’t fools and can see the consequence of these mad Auckland house prices is that within a generation the city will become a province of China unless something changes. The fact that Key blankly refuses to acknowledge the obvious makes him look ineffectual. And everyone can see that behind Key there is little of substance in his Cabinet.

    So while Key’s grip on the electorate has definitely been loosened, it’s not at all clear that people are ready to swing to the Oppo block just yet. That’s a threshold Labour and Little have not yet managed to climb over. (And of course the job of our media is to ensure they don’t.)

    • Draco T Bastard 2.1
      28 September 2016 at 12:20 pm

      The fact that Key blankly refuses to acknowledge the obvious makes him look ineffectual. And everyone can see that behind Key there is little of substance in his Cabinet.

      And, IMO, people are waking up to the fact that Key himself has no substance. Bit of a way to go before they’ll admit to themselves that they’ve been supporting an outright liar though.

  3. WILD KATIPO 3
    28 September 2016 at 9:53 am

    Well,..with the likes of Paula Bennet, Jerry Brownlee, Nick Smith , Judith Collins as your lieutenants , its no small wonder why Key was booed unceremoniously at a rugby league match and the LGBT festival.

    If one were to look back through the years of all the broken promises starting with things like this ,…
    http://www.stuff.co.nz/national/politics/10468960/Aroha-of-McGehan-Close-flees-NZ

    …see the anti worker legislation , the fiasco with the GSCB and SIS , Dirty Politics ( and the current investigation by the Ombudsmen ) the Christchurch EQC incompetence , Heath and Safety and worm farms/Pike River , the long , long list of lies such as this ,

    https://offkey.hackpad.com/Honest-John-NgiWfXTpnvD

    https://offkey.hackpad.com/ep/group/DTAANMceniq

    … the spying on behalf of the USA 5 eyes on Islands of the Pacific, China ( our biggest ‘ Free Trade Deal’ partner who sells us bad steel …) the Saudi sheep farmers and certain monetary sweeteners, … you know,… its really only been the fact people have been made paper millionaires because of a runaway out of control immigration policy creating a cheap labor market and the unregulated speculation in housing Key has encouraged ,… and now even that’s turning to shit as people wake up and see other Kiwis having to live in cars and garages…

    The housing crisis is the end for this govt , its been 9 long years of this do nothing , govt by deceit .

    You can only keep bending that rod of steel back and forth before it gets metal fatigue and snaps in the middle,… and these polls will continually start to reflect this as people worry about their paper wealth evaporating overnight when the bubble bursts and as they see their own kids locked forever out of home ownership in place of some landlord that doesn’t even live in this country.

    Welcome to the so called ‘ brighter future ‘ of Planet Key , … just don’t stay too long because the earthquakes are starting to rumble and the volcanoes are beginning to belch toxic fumes.

    • Colonial Viper 3.1
      28 September 2016 at 9:56 am

      The old “tide going out” on the government analysis is outdated IMO. We’ll know come Budget next year whether or not the NATs intend to get a fourth term. (They do.)

      • Lanthanide 3.1.1
        28 September 2016 at 10:10 am

        Oh yeah, I forgot about that tax cut bribe they have up their sleeve.

        But actually, the electorate may look at that, and decide “actually we’d rather have cheaper houses in Auckland through a state-sponsored building programme, not a tax cut”.

      • TIger Mountain 3.1.2
        28 September 2016 at 11:08 am

        what ray of sunshine…

        “there are few relations more bitter than those between former comrades”-quote from me, actually, originally applied to a section of the small NZ Marxist left, but it seems equally applicable to ex social democrats too

        ok, you are down on the NZ LP Mr viper, and fair enough for various reasons, but there are other things happening than parliamentary politics

        • Colonial Viper 3.1.2.1
          28 September 2016 at 11:13 am

          Yes indeed, there are.

        • rhinocrates 3.1.2.2
          28 September 2016 at 11:26 am

          William Congreve:

          “Heaven has no rage like love to hatred turned, Nor hell a fury like a woman scorned,” spoken by Zara in Act III, Scene VIII. The Mourning Bride (1697)

    • Draco T Bastard 3.2
      28 September 2016 at 12:29 pm

      The housing crisis is the end for this govt , its been 9 long years of this do nothing , govt by deceit .

      It’s not that they’ve done nothing. It’s that they’ve done nothing for the majority of people while heavily subsidising the rich from the pockets of the poor.

      • WILD KATIPO 3.2.1
        28 September 2016 at 1:41 pm

        Yes, another way of putting it , I guess. Fact is,… you cannot sustain this sort of abuse of large numbers of demographics ie: the growing numbers of the working poor , the homeless, those disillusioned with events that have precipitated people being locked out of home ownership,… it has a shelf life.

        And we are seeing not one big issue that’s bringing this govt down , but a long series of negative situations over the course of 9 years , many of which didn’t affect people directly or personally , however , … hitting people in their pockets and interfering with their plans for the future and denying them choices is a sure fire way to get people thinking ‘ just what is this govt all about ?’…and that’s the interface with a change of public mood and perceptions about any political party which starts to translate that sentiment in the polls.

        I suspect that we will see the end of this govt in a slow grinding down. And possibly , there will be a sure but steady incremental growth of the Lab/ Green MOU, and even more possibly , NZ First having quite a lot of leverage in creating a center left govt.

  4. dukeofurl 4
    28 September 2016 at 9:57 am

    The NY Times did a experiment. They did a poll of 700-800 people in Florida and then sent the same raw data to 4 other polling organisations to analyse.

    The same raw data let to a 5% spread in results from Trump +1 to Clinton +4!!

    http://www.nytimes.com/interactive/2016/09/20/upshot/the-error-the-polling-world-rarely-talks-about.html?_r=0

    You can see the differences just from methodology in this experiment, I think its mostly because the respondents dont match a model they have of voters.

    • Lanthanide 4.1
      28 September 2016 at 10:06 am

      Yip, all pollsters have to take the raw data and apply their secret sauce of “likely voter” and demographic skew, to create what they think is a reflection of the population.

      I’m not sure how sophisticated the likely voter and demographic modelling is that is used by the NZ polling companies. Certainly in the US they’re quite sophisticated, and that when you take the average of all the polls, the biases and errors should generally cancel themselves out.

      • Nic the NZer 4.1.1
        28 September 2016 at 8:18 pm

        I expect its very ‘sophisticated’. If it was not your poll would be subject to the vaguaries of presenting whatever result happens to fall out of the raw data.

        As for the average, whats the average of three public polls slanted towards the right and one private poll which is slanted left?

  5. Puckish Rogue 5
    28 September 2016 at 10:03 am

    http://www.kiwiblog.co.nz/

    That’s why (love him or loathe him) I follow David Farrars poll average as its up to date and he gives his methodology as well

    National: 44.3% 53 Seats
    Labour: 30.3% 37 seats
    Green: 12.4% 15 seats
    NZFirst 9.6% 12 seats

    and Act, Maori Party and Dunne on combined 2.7% of the vote bring in 4 seats

    • Paul 5.1
      28 September 2016 at 10:09 am

      The trolls arrive.
      Farrar told them what to say.

      • Colonial Viper 5.1.1
        28 September 2016 at 10:22 am

        That poll of polls actually reads pretty good for the possibilities of a Labour/GR/NZF coalition.

        Another 1% to 2% move up away from NAT towards the LAB/GR block and Winston will very likely go with LAB/GR.

        • Puckish Rogue 5.1.1.1
          28 September 2016 at 10:32 am

          Don’t mind Paul, he probably thinks he’s getting one over Thatcher 🙂

        • DISTURBED 5.1.1.2
          29 September 2016 at 2:33 pm

          Yes Winston will go with Labour/Green for sure.

          No one even Winston wants a bar of this toxic rogue Junta, they call National now!

          They are anything other than National.

          Simply they are Just property agent’s for overseas investors.

          Not caring about our Kiwi population at all.

          But are something like; they favour “Genocide” calling us all “to drugged up to employ” what the hell did they say that for????

          I don’t take any drugs.

    • mpledger 5.2
      28 September 2016 at 11:29 am

      I’ve been polled by Curia – they asked for me by name and the interviewer was partisan so I am somewhat dubious of their methodology and their ethics.

      • rhinocrates 5.2.1
        28 September 2016 at 12:44 pm

        Curia is notorious for push polling.

      • DISTURBED 5.2.2
        29 September 2016 at 2:37 pm

        Yes me too, I was asked several real strange questions not relating to elections and when I answered one specific question they abruptly said no we don’t need you for this poll thank you?????
        I think they first are seeking a particular social set of questions then target them probably more likely to vote national and I didn’t fit the mould.

        • te reo putake 5.2.2.1
          29 September 2016 at 2:47 pm

          The answer is actually pretty dull. The surveys look to balance the responders, so they don’t for example, get too many Nats or not enough Greens or too many uni students and not enough pensioners. So they ask filter questions and if you fall into a category that already has its quota filled, they end the call at that point.

  6. The Real Matthew 6
    28 September 2016 at 10:06 am

    Looks like a rogue poll to me

    • Paul 6.1
      28 September 2016 at 10:09 am

      The trolls arrive.
      Farrar told them what to say.

    • mpledger 6.2
      28 September 2016 at 11:30 am

      It’s hard to be a rogue poll when it’s consistent with other polls.

      • Matthew 6.2.1
        28 September 2016 at 2:53 pm

        You’re assuming that TRM even knows what a rogue poll actually is, and understands concepts like “on-trend” and “off-trend.” I imagine he’s just picked up the phrase from Farrar or something.

      • The Real Matthew 6.2.2
        28 September 2016 at 3:31 pm

        What other polls is this consistent with? Only Labour’s trumped up internal polling.

        Not consistent with the poll showing Labour at 26% is it?

        • Matthew Whitehead 6.2.2.1
          28 September 2016 at 4:50 pm

          Mate, if you commission an internal poll, you don’t do it to inflate your numbers- in fact, that’s far more likely for a public poll.

          When you commission an internal poll you’re looking to find the weaknesses in both your strategy and your opponent’s strategy, which is why you keep them private. Internals aren’t useful if they’re inflated. Arguably you want them to be as pessimistic as realistically possible.

  7. Fisiani 7
    28 September 2016 at 10:20 am

    To be the PM 2018-20 Little simply has to make Winston PM for 17-18 and exclude Greens from Cabinet. Simple deal

  8. TIger Mountain 8
    28 September 2016 at 10:59 am

    housing will ultimately do for our glorious leader despite the nuisance level polls in this country

    a Roskill by election will be a suitable trial for the G/L MoU and if that goes well it could be extended to the general election and Epsom bumboy and Hairdo territory, and if the MP defects too; things will get really interesting

  9. Autonomouse 9
    28 September 2016 at 11:40 am

    Given the rife apathy in regard to vote casting, what I’d really like to see is the addition of a couple of questions within polls, namely:

    1. Did you vote in the last election? (Yes/No)
    2. On the balance of probabilities, are you likely to vote in the 2017 General Election? (Yes/No/Maybe)
    3. If you did vote in the last election, will you be voting for the same political party during the 2017 election? (Yes/No/Maybe)

    Surely such questions would give a better insight into the “Missing Million”, enable an understanding as to the number of “Swing Voters”, and show current polling compared with that of the previous election taking account those that statistically matter (being those that actually vote).

    Are such questions incorporated within any of the publicly available polls?

    • Matthew 9.1
      28 September 2016 at 2:57 pm

      Most NZ-based pollsters already exclude anyone who doesn’t describe themselves as a “likely voter.” It’s a rubbish methodology, instead they should ask them to rate on a scale of how likely they are to vote and weight responses based on that.

  10. Takere 10
    28 September 2016 at 12:32 pm

    Putting aside the BS methodology;
    Small samples
    Landlines only
    Time of day people are called
    Favourable “Polling-Tree”
    And the questions asked prior to the actual Polling Question of who’d they vote for if there was an election ….
    Its the impact of the perception impressed upon people when its presented to them as a first past the post proposition in lame stream media.
    So it is important that the polls & polling is contested every time the next one is released as to try and encourage people to “think” first before believing everything they read? Thats a major fault of many kiwi’s. Thinking before they regurgitate what they’ve just read or heard from someone….

  11. James 11
    28 September 2016 at 12:48 pm

    The Roymorgan is all over the place – but hey.

    Lets accept it at face value. In which case – good on you Labour – well done.

    Lets also accept the next few from them at face value as well.

    I can see the headlines for the next election now:

    https://fmacskasy2.files.wordpress.com/2012/09/nz-first-national-coalition-11-12-96.jpg

    • Lanthanide 11.1
      28 September 2016 at 1:57 pm

      Or, lets apply our brains and judge the situation as a whole.

      In which case – good on you Labour – well done.

    • Matthew 11.2
      28 September 2016 at 2:58 pm

      Right, because there’s no chance anyone in NZ First actually remembers how badly that worked out for them, lol.

      • Phil 11.2.1
        29 September 2016 at 9:12 am

        Aside from Winston, I doubt there is anyone in the party today who was around at the time to remember how badly it went for them.

  12. ianmac 12
    28 September 2016 at 3:46 pm

    Anyone noticed this on Stuff Politics?
    “Labour will find little to complain about in the latest political poll after it gained 8 points to reach 33.5 per cent.

    The Roy Morgan poll has National down 4.5 points taking its support to the lowest level in three years at 41.5 per cent.

    Earlier this month Labour leader Andrew Little wrote off a One News-Colmar Brunton poll as not “credible” and “bogus” after Labour slumped to 26 per cent.

    It came at the same time the Opposition’s own poll, carried out by UMR, had Labour at 31 per cent, National on 40 per cent, the Greens on 14 per cent and NZ First on 11 per cent.”
    http://www.stuff.co.nz/national/politics/84729941/labour-up-8-per-cent-in-latest-roy-morgan-poll-but-nz-first-still-holds-the-power

  13. Matthew Whitehead 13
    28 September 2016 at 4:53 pm

    BTW, flash polls are validating your assumptions on Hillary. Both scientific polls released so far have her winning the debate convincingly, and the only focus group running during the debate that had her win was MSNBC, which had a comparatively small group where trump won by one vote.

  14. Incognito 14
    28 September 2016 at 10:52 pm

    Or 1 in 20 polls is just plain wrong?

    There are two ways I can argue the toss:

    1) All polls are wrong.
    2) All polls are right.

    Polls are just small samples from a much larger pool – statisticians talk about “populations” rather than pools – and that pool is all eligible voters, or perhaps all eligible voters who may actually bother to vote. In any case, many (?) voters are still to decide how they’ll vote – and they will be influenced by (the) polls among other things. Therefore, all polls are wrong regardless.

    All polls include a margin of error (MOE) usually at the 95% confidence level. This MOE is for the mid-point, i.e. for 50% of the sample giving a certain result/answer. For example, if 1,000 voters are polled and 500 (50%) say they’ll vote for Labour the MOE is 3.1%. In other words, if all voters would have voted then & there the vote result (the so-called “true value”) will have a 95% likelihood of being within the predicted range of (50 ± 3.1)%. It is less likely, but not impossible, that the result will be outside this range. It is even possible, but highly unlikely based on that one single poll, that less than 40% of all voters do indeed vote for Labour. The chance of that occurring is one in several billion but still not a zero chance. Thus, all polls are right 😉

    Statistics is all about chance, probability, likelihood, statistical power, etc.; there is no right or wrong.

    Disclaimer: I’m a statistical amateur, not a professional statistician, and the likelihood that I’m wrong is high

    • dukeofurl 14.1
      29 September 2016 at 8:12 am

      The issue is using statistical terms and putting them in everyday terms that people understand AND are still correct

      Example is the flooding or weather events, described as 1 in a 100 year event. Most people use that as a ‘predictor’ and think only one every hundred years which isnt right.
      Its really a probability of 1% ( thats where the 1 in 100 comes from), so that every year there is a 1% chance. This can mean you could have two 1% storms or floods 5 years apart. Added to that is the difficulty in knowing the exact rainfall or flood height till well after so one storm might be 2% ( 1 in 50) and the other 1%.

      Polling statistical terms still seem to be in a flux. The 5% confidence level or 1 in 20 I would see as taking the last 20 polls and removing the most extreme . Not really possible in practical terms but I think some pollers have quietly dropped a poll when the results are to far out of whack with every one else

  16. DISTURBED 16
    29 September 2016 at 2:22 pm

    Good wrap that Ben,

    Yes with Phil Twyford at the helm as “Twyford to chair Labour’s election campaign”
    they cant go wrong here, and it shows in this poll the approval of Phil’s hard work constantly on National’s back about homeless that got everyone really shocked finally, he is a very engaging individual, – good on him there is hope there.

The server will be getting hardware changes this evening starting at 10pm NZDT.
The site will be off line for some hours.