On the perils of polls

43 comments on “On the perils of polls ”

• Comments are now closed

1. Steve Withers 1

   30 May 2008 at 11:42 am

   Poll results are also grossly distorted if the number of people undecided is not sampled and reported. A result of 56% of decided voters where 25% of all voters weren’t decided is a very different beast to the same result where only 5% are undecided.

   56% of 75% is barely more than 42% support among the whole sample, assuming undecided voters do actually vote. I know they often do not vote.
2. Bryan Spondre 2

   30 May 2008 at 11:47 am

   I assume there is a link between this post on the perils of polling and the expanding gap between National and Labour ?

   PS I seem to be back out of moderation 🙂
3. Steve Pierson 3

   30 May 2008 at 11:51 am

   In the US, they try to sample probable voters (more important there because the turnout is only 50%). It all gets very complicated. There are big systematic problems with polling and the corrections that are attempted are problematic in themselves.

   In their final polls last election, 4 of the 5 public polling companies had Labour under their their actual result.

   Of course, that could always have been due to a late surge from Labour or it could be a systematic tendency to underpoll Laobur support.
4. Steve Pierson 4

   30 May 2008 at 11:53 am

   Bryan. That’s the most stupid thing you’ve ever written, and that’s a pretty high bar.

   You’ve got a graph right on this page of the past 22 poll results and it shows a marginal narrowing of the gap.

   Comments like that are an insult to everyone’s intelligence
5. sean14 5

   30 May 2008 at 12:02 pm

   Yawn. Bring on election day.
6. Phil 6

   30 May 2008 at 12:13 pm

   “… polling companies use ‘witches brews’ of formulas to balance the demographics of their samples to that of the general population. It can raise its own problems but, apparently, polling companies in New Zealand don’t even do that.”

   This is simply not true. AC-N certainly does, Roy Morgan, as far as I am aware, do too. Not sure about CB.

   Market research compaines in NZ are held to a very high standard by the Market Research Society http://www.mrsnz.org.nz/ and I personally think that a guest post from someone there might prove illuminating.
7. Monty 7

   30 May 2008 at 12:19 pm

   Hummm – you lefties sure think that there are a lot of rogue polls – but then you can relax because inevitably the subsequent polls confirm that Labour is buggered. The only polls you do seem to like are the ones us righties refer to a s a “dead cat bounce”.

   You need to understand that the electorate is comfortable with National winnning about 50% of the vote on polling day. The electorate is also saying that Labour need to go. The budget has been no use and I think there will be more polls before the election that have Labour below 30%
8. Steve Pierson 8

   30 May 2008 at 12:20 pm

   Phil. Happy to be corrected on that. Guess Curia’s not in that league.

   It doesn’t detract from the main point, which is movements in polls are more likely normal variation or a sampling problem than a change in the population.

   It’s intersting to look at that trend line. Support appears remarkably consistent.
9. DS 9

   30 May 2008 at 12:37 pm

   “You need to understand that the electorate is comfortable with National winnning about 50% of the vote on polling day. The electorate is also saying that Labour need to go. The budget has been no use and I think there will be more polls before the election that have Labour below 30%”

   If National couldn’t crack 50% in 1990 against what was at the time the most hated incumbent government of the postwar era, they aren’t going to do it in 2008, polls or no polls. See Labour in 2002 for how easily, say, 56% poll ratings can be turned into an actual election night figure of, say, 41%.

   You Tory types had also better be hoping like hell that you can get 47% or so. Anything less than that and you suddenly get shafted by the fact that you’ve long since cannibalised your coalition partners.
10. Bryan Spondre 10

    30 May 2008 at 12:59 pm

    SP: “You’ve got a graph right on this page of the past 22 poll results” presume you mean the chart titled “Rogue Polls” ?
11. Lew 11

    30 May 2008 at 1:24 pm

    DS: “You Tory types had also better be hoping like hell that you can get 47% or so. Anything less than that and you suddenly get shafted by the fact that you’ve long since cannibalised your coalition partners.”

    Nasty rhetoric aside, this is the major issue. Broadly there are two possible strategies to play here.

    1. National go alone (for card players). Requires 50% less however many seats ACT gets to guarantee a government. Also comes with bragging rights of winning a clear plurality or perhaps a majority, but that’s more valuable before an election than after. Clearly the best strategy if it’s successful, but National risks complete failure if they fail to get 50% with ACT.

    2. National curry favour with NZ First and United Future, and try to raise both parties’ profiles and party votes, not that there’s any love lost with NZF. This is a strategy which suits Key’s `Labour Lite’ sort of image: a consultative, collaborative politician who understands the value of consensus and most importantly understands MMP politics. It’s a valuable strategy to choose publicly well before the election, since those characteristics could win over some Labour voters who don’t much like Labour policy but dislike the image of National as a big brash winner-takes-all party: plenty of women fall into this category so there’s a synergy with Key’s leadership appeal too. It’s a risky strategy, though, since it will require National to declare policy soon, in detail, and begin public negotiations with other parties, and that makes them a bigger target. Also, if National loses some of its own electorate to UF and NZF, who then decide to go with Labour, they lose it all.

    If I were John Key’s strategist I’d take option 2. I think National are positioning for 1. Who knew that being burnt in 1996 would still shape their core political strategy 12 years later?

    L
12. Tane 12

    30 May 2008 at 1:26 pm

    How’s that rhetoric nasty Lew?
13. Lew 13

    30 May 2008 at 1:28 pm

    Tane: `You Tory types’ `hope like hell’ `shafted’ `cannibalised’. I’m not saying it’s offensive, just uncivil.

    L
14. Tane 14

    30 May 2008 at 1:37 pm

    Well, politics ain’t civil Lew. No need to have a go at people unless they’re out of line.
15. Steve Withers 15

    30 May 2008 at 1:41 pm

    Steve Levine and Nigel Roberts studies over the years have shown that no matter how you slice the pie, the centre-right just does not have majority support among voters. So “partners” there are of little use. National would ned to move BACK to the centre to win (and keep) power. It’s debatable they are prepared to do more than pretend as some of their richest supporters want some of the more extreme (relative to voter tolerance) policies enacted.

    MMP has exposed the real composition of the electorate, while First Past the Post masked it and effectively prevented the fragmented ‘non-centre-right’ majority from asserting itself term after term UNLESS they backed one party or a third party (NZ Party 1984) split the centre-right vote and allowed them to come through the middle.

    NZ First is essentially the chunk of National that jumped off the boat when they lurched from the Centre(ish) to starboard (right) from 1990 onward.

    Labour also fragmented, but most of the pieces (ACT excepted) stayed to the left of National. Labour has effectively owned the “centre” with the 1996 anomaly of Peters going with National while his supporters wanted Labour, as a ‘rogue’.

    For National to get 50%+ of the vote in this year’s election would be a first in over 50 years. I’d be surprised if it happened. But anything can happen.
16. Lew 16

    30 May 2008 at 1:49 pm

    Tane: I wasn’t having a go at DS, it’s perfectly reasonable for him to say it. Easy there.

    SW: Yeah, I was thinking of Roberts and Levine and the impact of MMP when writing this up, too.

    Any of `you Tory types’ want to give an opinion as to why Key might be choosing 1 over 2 when 2 appears more likely to succeed? I’m curious whether he has your support.

    L
17. Robinsod 17

    30 May 2008 at 2:00 pm

    Um Lew – 50%? You forgot about the overhang which is generally considered to broadly favour the left. Apart from that you don’t seem to be too badly wrong…
18. Lew 18

    30 May 2008 at 2:04 pm

    Sod: Round figures; you take my meaning. I’ll take that as your one random compliment for the month of May.

    L
19. Robinsod 19

    30 May 2008 at 2:10 pm

    you take my meaning

    Is that in the imperative? I mean do I get a choice or am I obliged to take your meaning?

    [becoming tiresome, ‘sod. SP]
20. George Darroch 20

    30 May 2008 at 2:30 pm

    Having worked on the telephones for a research polling company, I can tell you that the sample of people who will do your polls (people who are at home, willing to answer, have a landline) is not representative of the general population. It’s a difficult issue to address, but all good research acknowledges the limitations of the data – bad research doesn’t.
21. Steve Pierson 21

    30 May 2008 at 2:55 pm

    And we all know what kind of quaility research you get in newspapers.

    Audrey Young’s piece on the Herald Digipoll takes the sample of 1200 and cuts it up by various demographics. All of which have much larger margins of error than the original sample.

    For example when you ask ‘how many 18-24 year olds are in my sample’ and then, ‘how many of those support the Greens’ you’re down to really small nubmers and the margin of error is huge.

    but you wouldn’t know it from her article which has figures stated down to one decimal point as if the poll can be that exact.
22. jafapete 22

    30 May 2008 at 3:12 pm

    Good post.

    To attempt an answer to Lew’s question, I’d say that the Nats are going for 2, but half-heartedly. But that’s okay, they figure, as it is all about what they can negotiate after the election. The real test will be whether they are willing to humiliate themselves by, say, making Winston Minister of Foreign Affairs. (Should they get most votes and first shot at forming the government.)

    Interestingly, they don’t seem to have worked out that they may need the Maori Party to abstain on C&S, or, more probably, what they need to do in the meantime to get that degree of MP support. As Brian Rudman pointed out earlier this week, last weekend they missed their best chance to make up for Bastion Point. Stupid, but not unexpected.

    Steve Withers: I think you’ll find that, of those RTS and undecideds who finish up voting, the voting patterns are usually close to those of the decideds who answer the poll. But not all the time.

    On a small technical point, not weighting the data need not necessarily mean “their chances of getting a rogue poll are that much stronger.” Are you referring to pre-stratification or post hoc weighting, or some combination?

    On weighting, the marketing companies that Phiul refers to are using the data for different purposes than political polling – often to gauge tastes or the impact of campaigns in a particular demographic segment. I’ve long since overcome my aversion to weighted data, partly because people don’t take into account over-representation of particular groups even if that caveat is carefully made clear at the outset. (Was it in this case?) But it does get a little dodgy when one is using weighted data in multivariate analysis.
23. jafapete 23

    30 May 2008 at 3:18 pm

    re Steve Pierson’s 2.55pm comment on Young’s analysis in the Herald.

    Exactly right, even in respect of the main numbers, never mind the cross-tabulations. Polling is NOT such a precise science that you can say, “Labour has moved down one point to 36.2 per cent but National has also moved down fractionally, by 0.6 to 51.5.”

    That’s just so silly.
24. Lew 24

    30 May 2008 at 4:06 pm

    jafapete: Yeah, implicit in my scenario is a false dualism, thanks for your response which brings this to light. Thanks also for your comments about polling (the nominal topic of this thread).

    On further reflection it seems National could be wisest to play a mixed strategy which begins as 1 and moves toward 2 gradually if or when the polls look like 1 is unachievable. National potentially are in the driver’s seat here, as long as they can come up with enough good robust defensible policy to hang their symbolic messages on, before the tipping point.

    Incidentally I think Labour’s best strategy is to take a leaf from National’s book and play a defensive campaign criticising their policy and message, letting government policy speak for itself. Like the Crusaders – play tight, dominate set piece and 50/50 ball, and most importantly: punish mistakes.

    Not that I’m a Crusaders fan, but their style of play gets results.

    L

    Captcha: `greater smart’.
25. Dave 25

    30 May 2008 at 4:49 pm

    It can raise its own problems but, apparently, polling companies in New Zealand don?t even do that.?

    This is simply not true. AC-N certainly does, Roy Morgan, as far as I am aware, do too. Not sure about CB.
    Colmar Brunton does, too.It monitors demographics to get an accurate sample
26. Andrew Bannister 26

    30 May 2008 at 6:05 pm

    A good post that illustrates a good point. However, I would like to point out a couple of small but important errors.

    1) You say “that is, one in 20 polls will be out from reality” and “One in twenty polls is a rogue poll “. That should really be “that is, one in 20 polls may reasonably be expected to be out from reality” and “One in twenty polls can reasonably be expected to be a rogue poll”. The stated rogueness isn’t a given. It is possible (unlikely, granted) that if you take 20 polls, 19 are out by more than 3% and only one falls within that margin of error.

    2) You say “That’s the Fairfax poll showing a 27% gap. Look how far it falls outside the polls before and since; an obvious rogue”. It is actually a probable rogue.

    Sorry to be a pedant, but I think these important distinctions.
27. Occasional Observer 27

    30 May 2008 at 6:52 pm

    It wouldn’t surprise me if Labour pass a law before the election giving the Prime Minister the right to cancel the election result if she thinks it is just a rogue poll.

    Enter a new expression into the Standard’s lexicon: the “rogue election”. When John Key’s National Party thrashes Labour by twenty points, even AFTER Labour has done everything in its legislative power to screw the electoral scrum, the Standard will claim the election as rogue.
28. Daveo 28

    30 May 2008 at 7:00 pm

    I thought it was National that was having trouble accepting they lost the last election? Seriously OO, your comments are getting more delusional by the day. Nine years in the cold must be tough.
29. AncientGeek 29

    30 May 2008 at 7:06 pm

    There are two types of demographic weighting used, from what I understand. One is to get the right people on the phone, and for them to answer. That is very hard to get a demographic that even approximates the population. A lot of people either don’t have land-line, a lot like me don’t have phone number listings, and a lot of people won’t answer.

    From phone canvassing you always get kinds of distributions shown in the curia poll. A lot older than the electorate, usually more affluent, and much more technophobic (ie don’t use caller id or cellphones). People who answer are usually more credulous as well. Ask any telemarketer – if you can pull them in on the first couple of questions, then they’re a born sucker. The canny ones politely hang up and tell you nothing.

    In other words the sample self selects according to who you can reach and who will answer.

    The next trick is to ‘adjust’ what data you do get to the demographics. That is where it gets tricky. If you have say 400 elderly and say 50 20-30’s. Then you weight down the elderly, and weight up the 50 conservative technophobic suckers with phones you caught in the younger age group. Since they pretty well all came from the equivalent of the North Shore and not from Mangere, you just multiplied a non-representive demographic.

    Of course we’re not counting the fact that a lot people simply haven’t thought about who they’d vote for this far out. They’re likely to say anything to get rid of you, especially when you catch them while they’re watching their favorite soap opera. Try targeting certain groups when Coro Street is running

    The whole procedure sucks on any scientific basis. I find it a indictment on the education system that anyone takes them seriously. But I suppose the media really need to fill their headlines.

    Good post Steve
30. AncientGeek 30

    30 May 2008 at 7:15 pm

    I suppose the best way of describing polls is that they are all rogues that get more accurate in the last few weeks as more people are prepared to answer.

    They’re still inaccurate at election day, but a lot less so than 5 months out.
31. bill brown 31

    30 May 2008 at 7:31 pm

    Espiner on polls:

    http://stuff.co.nz//blogs/politics/2008/05/30/all-eyes-on-the-polls-after-bad-week-for-labour
32. milo 32

    30 May 2008 at 9:30 pm

    Actually, DPF has it right in reporting his poll of polls. Combining all the polls increases the sample size (reducing the confidence interval), and also allows the sampling biases from different survey methods to balance each other. It’s the best practice method. You can also see it in play in the commentary on the US Presidential Election.

    Also, I’d be careful about the idea of a “rogue”. It sort of implies a mutant ogre escaping from the research company and laying waste the landscape. In fact, “rogue” polls are just polls that are a little less accurate than most. Being more than 3% out doesn’t mean that you are 10% out; chances are you are just 4% out instead.
33. Lew 33

    31 May 2008 at 12:15 am

    Occasional Observer: If it happened, I think you’d be entirely justified in waging armed rebellion against the government.

    If it doesn’t, and it won’t, your only legitimate recourse is to the electoral system, like everyone else.

    But you’re welcome to your paranoid delusions in the mean time.

    L
34. AncientGeek 34

    31 May 2008 at 12:39 am

    milo: It helps with increasing the sample size.

    However it doesn’t help with the underlying selection bias. As far as I’m aware, all public polls in NZ are done using landlines and scripts. There is an inherent bias just in that.

    If different polls used different techniques, then it’d be more interesting.

    You get markedly different results just between phone polling and door canvassing in the same areas. Within an electorate, you get major result differences between areas that are separated only by a few streets. You get quite different results depending on the times you call.

    Since the methodologies of the polls aren’t published, we simply don’t know how much difference there is between the polls. If they were following similar methodologies (as I suspect) then you get more accurate assessments of the same methodological flaws.

    Frankly the most useful information that the polling companies could publish is
    – How many targets did they fail to contact
    – How many people refused to answer
35. milo 35

    31 May 2008 at 10:57 am

    Ancient Greek – I wouldn’t want to suggest averaging across polls
    gets rids of all bias. But it does reduce it.
36. Ari 36

    31 May 2008 at 11:06 am

    AG- there’s a lot of problems with aggregating polls. I agree with you on publishing misses though- they need to be included.

    For example, do you weight the aggregate by the number of people in the poll? The accuracy rating of the poll? Do you include rogue polls that are out by more than the usual accuracy rating?

    That’s even ignoring issues like when you cut off your aggregator- after a certain amount of time goes by, it’s not longer worth including an old poll in an aggregate because the reasons people polled a certain way are outdated and their opinion may have changed.

    A good, rigorous aggregate that had access to the unreleased data from polling companies, and compensated for flaws in their respective methods would be totally awesome. Someone like Davey going through and aggregating by hand strikes me as likely to be just as bad as the polls themselves, as there’s likely to be no weighting.
37. Jimmy 37

    31 May 2008 at 11:33 am

    There’s heaps on US polling & aggregating and whatnot here:

    http://www.electoral-vote.com

    Its all terribly interesting.
38. Steve Pierson 38

    31 May 2008 at 3:54 pm

    The more technical arguments are interesting and I appreciate Andew Bannister’s points on statistical language. I’m aware of the distinctions, just don’t think they’re all that important in the context of a post on a political blog for a general audience.
39. jafapete 39

    31 May 2008 at 4:25 pm

    Ancient Geek,
    You illustrate well (7.06pm yesterday) how weighting works, but your numbers do not do the practice justice. If you have a decent sample size (I wouldn’t trust a national opinion poll with fewer than 1000 respondents), then the variance from the population shouldn’t be that great, and certainly not in the order that you use to illustrate.

    I’ve found that weighting brings the results very close to the population parameters where these are known from censuses and the like.

    We have a well constructed election poll based on good methods, but it was taken mostly *after* the election. It is a postal survey conducted by academics, and called the NZ Election Study. Sadly, FRST stopped funding it before the last election, and it is now run in diminished form. Details at http://www.nzes.org/

    The VUW survey is done on telephone and is pretty crappy — I was a respondent at the last election. It is taken just a few days before the election, which is the main reason why the VUW people are able to claim it is very accurate.
40. andydoanx 40

    4 November 2008 at 4:35 am

    what if in elector list there is name which actually the candidate has passed away?

    is still valid to made substitution?

    Thanks
41. RedBack 41

    4 November 2008 at 7:07 am

    Steve well done on bringing up the pesky margin of error.
    As we all know these polls are comissioned by various media organisations to suit their own front pages. The questions are usually leading and will enable the publication to arrive at the result they were hoping for. I point folks to the Heralds highly intellectual poll question ‘Which politican would frighten children the most on Halloween’ While some polls may carry some truth they should never be used as the be all and end all of predicting an election result. As an analyst myself I can gaurentee you that the hang up rate pollsters encounter is roughly 90- 95%. This means you are often left trying to collate results with the shall we say socially angry. Most of whom have an ultra conservative axe to grind and would think their opinion really matters when confronted with a question offered such as the Heralds pointless character assesment polls. The danger is when these polls are paraded as a rock solid prediction with no margin of error published or more importantly undecided.

Links to post

1. Chris Hipkins - Candidate’s Blog » Blog Archive » The Budget and the Pollsters: posted on 30 May 2008 at 12:44 pm
2. Polling companies confirm reputations at The Standard 2.01: posted on 3 June 2008 at 12:02 pm

• Comments are now closed

• Log in
• Donate
• Contribute post

• Articles RSS