Written By: - Date published: 11:28 am, May 30th, 2008 - 41 comments
Categories: Media, polls -
Tags: curia
We’re used to hearing that a poll has a margin of error but what does it mean? A margin of error of 3% doesn’t mean the poll’s numbers are definitely within 3% of the ‘real’ numbers, it means there is a 95% chance they are within 3% of those numbers; that is, one in 20 polls will be out from reality by something greater than 3%. One in twenty polls is a rogue poll and there is nothing that a polling company can do to prevent that. See those Xs on the graph of the last 22 polls below? That’s the Fairfax poll showing a 27% gap. Look how far it falls outside the polls before and since; an obvious rogue.
But polls can also be out if the sample isn’t random. Random doesn’t just mean the first 1000 people who you get to answer. It means that your sample is not different from the general population. Let’s take a bad poll to see how this can go wrong – David Farrar’s recent poll for Family First on smacking.
In the general population, 17% of people are over 60, 30% of the respondents to Farrar’s poll were over 60. 37% of kiwi families have children at home, in Farrar’s poll only 22% of them did. The other demographic data is also out. This means that group of people Farrar sampled is not a real sample of New Zealand and the results may be wrong over and above the margin of error that is always there. Farrar’s poll shows reasonable support for smacking but that support is especially strong in the over-sampled demographics (old people and those without kids).
Now, in America, polling companies use ‘witches brews’ of formulas to balance the demographics of their samples to that of the general population. It can raise its own problems but, apparently, polling companies in New Zealand don’t even do that. Meaning their chances of getting a rogue poll are that much stronger. And don’t forget: polls are done by calling landlines, not everyone has a landline and 70% of people refuse to take part in polls that means the sample one gets in any poll is attitudinally different from the Kiwi population in general.
On top of all this, not all polling companies are created equal. In New Zealand, Colmar Brunton is notoriously inaccurate in its political polling, leaning about 5% to National, while Roy Morgan is the best on the major parties but over-polls the Greens. That comes down to methodology and, some have suggested, bias in polling companies. At any rate, polls are likely to be well out from the true numbers. How much were the final polls before the last election out in total, from reality?
What does all this mean? Individual polls may not reflect reality and a movement in results between polls, especially in the absence of a major political event (eg Orewa I), is more likely to result from normal variation or a problem with the polls than from a change in the real support levels for parties.
So, next time you see a 27% gap when there was a 15% one before, don’t get too excited.
Facebook
Twitter
Scoopit!
AG- there’s a lot of problems with aggregating polls. I agree with you on publishing misses though- they need to be included.
For example, do you weight the aggregate by the number of people in the poll? The accuracy rating of the poll? Do you include rogue polls that are out by more than the usual accuracy rating?
That’s even ignoring issues like when you cut off your aggregator- after a certain amount of time goes by, it’s not longer worth including an old poll in an aggregate because the reasons people polled a certain way are outdated and their opinion may have changed.
A good, rigorous aggregate that had access to the unreleased data from polling companies, and compensated for flaws in their respective methods would be totally awesome. Someone like Davey going through and aggregating by hand strikes me as likely to be just as bad as the polls themselves, as there’s likely to be no weighting.
There’s heaps on US polling & aggregating and whatnot here:
http://www.electoral-vote.com
Its all terribly interesting.
The more technical arguments are interesting and I appreciate Andew Bannister’s points on statistical language. I’m aware of the distinctions, just don’t think they’re all that important in the context of a post on a political blog for a general audience.
Ancient Geek,
You illustrate well (7.06pm yesterday) how weighting works, but your numbers do not do the practice justice. If you have a decent sample size (I wouldn’t trust a national opinion poll with fewer than 1000 respondents), then the variance from the population shouldn’t be that great, and certainly not in the order that you use to illustrate.
I’ve found that weighting brings the results very close to the population parameters where these are known from censuses and the like.
We have a well constructed election poll based on good methods, but it was taken mostly *after* the election. It is a postal survey conducted by academics, and called the NZ Election Study. Sadly, FRST stopped funding it before the last election, and it is now run in diminished form. Details at http://www.nzes.org/
The VUW survey is done on telephone and is pretty crappy — I was a respondent at the last election. It is taken just a few days before the election, which is the main reason why the VUW people are able to claim it is very accurate.
what if in elector list there is name which actually the candidate has passed away?
is still valid to made substitution?
Thanks
Steve well done on bringing up the pesky margin of error.
As we all know these polls are comissioned by various media organisations to suit their own front pages. The questions are usually leading and will enable the publication to arrive at the result they were hoping for. I point folks to the Heralds highly intellectual poll question ‘Which politican would frighten children the most on Halloween’ While some polls may carry some truth they should never be used as the be all and end all of predicting an election result. As an analyst myself I can gaurentee you that the hang up rate pollsters encounter is roughly 90- 95%. This means you are often left trying to collate results with the shall we say socially angry. Most of whom have an ultra conservative axe to grind and would think their opinion really matters when confronted with a question offered such as the Heralds pointless character assesment polls. The danger is when these polls are paraded as a rock solid prediction with no margin of error published or more importantly undecided.