web analytics

Pretend polling

Written By: - Date published: 10:03 am, November 16th, 2007 - 32 comments
Categories: polls - Tags:

Dodgy online polls seem to be becoming a staple of political reporting here in New Zealand. The story earlier this week on poll “hacking” sent me in search of an insightful opinion I’d heard previously.

So today, we have a guest author on The Standard – BPGP, on “Pretend polling”.

Pretend Polling
BPGP

So for the second time this year a newspaper publisher has made a story out of the fictions that are their online poll results. The first was a piece earlier this year in the Dominion Post alleging Parliamentary staff had skewed their online preferred PM poll to show Clark was ahead of Key. And now we have the Herald shrieking about how a teenaged hacker skewed their online poll to show that readers didn’t think New Zealand was “becoming a less free and democratic country”, despite the Herald’s concerted efforts to convince its readers we’re heading for a Stalinist dystopia.

Both these stories demonstrate just how bogus online polls really are, and it makes me doubt their creators have ever heard of the terms “validity” or “sampling”. It is interesting though, that despite most online newspaper polls being regularly quoted as if they were the gospel truth (at least when their outcomes show opinions advantageous to their publishers’ editorial interests), these two polls are miraculously exposed as skewed.

Sometimes publishers are generous enough to mention that the polls “aren’t scientific” & a discrete way of saying they have no validity. But invariably the next sentence will go on to quote the results as if they do have meaning that can be generalized to the greater population. After all, why would you bother to run a poll if it was actually completely meaningless?

Well, there are several reasons, but few of them to do with illuminating public opinion.

One is that online polls are useful for giving the impression that publishers interact with their audiences and are genuinely interested in listening. You might say that polls help media outlets to gauge the opinion of their audiences, but in reality they aren’t so foolish as to alter their editorial stances on the basis of such flawed data (papers like the Herald use a more sound, random sampling procedure to invite clickers to participate in a Readers’ Survey for that). Online polls are also a useful gimmick for keeping up online readers’ interest and so contributing to the maintenance of hit rates.

But the main reason for using online polling is to create an impression of popular support for the outlet’s editorial stance. If you were really cynical you might think that poll commissioners just pump the data in whichever way they want it, but in reality it’s quite unnecessary to engage in such risky manipulation. There are other ways. Like priming readers with skewed reporting on the issue, such that so long as they rely on that outlet for their information on a particular issue, they are bound to come to similar conclusions as those promoted by the outlet’s reporting.

Another way is to use priming questions. The Section 59 “debate” was a gold plated example, asking questions like “Should parents have the right to discipline their children?” when in fact the bill had nothing to do with “rights to discipline”. And then there’s misrepresenting what the poll question actually meant, as in saying that because a majority understandably agree that “parents have the right to discipline their children”, they therefore are opposed to Section 59. Yet another way is to massage the response options. Typically, very complex questions are forced into Yes/No dichotomies, when respondents may in fact have more moderate or complicated responses.

Similarly, you can alter a result just by the number of response options you allow. For example, you could ask “Do you support the EFB – Yes or No?” and get 60% for and 40% against. But if you ask “Do you support the EFB – Yes, No, or Yes but with some changes?” you might get 30% Yes, 40% No, and 30% Yes but with some changes. These results of course, would then be reported as “70% percent opposed to EFB in current form”, when the same question with just a Yes/No format would have provided an unhelpful result (for the Herald at least) showing majority support for the EFB.

The Herald and Dominion Post articles point to another, user driven form of manipulating the results – multiple voting. No matter what newspapers claim about security to measures to stop such behaviour, they will always be vulnerable to it. Cookies and IP tracking to monitor voting behaviour can easily be circumvented with SSL-flushing between votes and freeware IP masking software. A little bit of script to automate the vote-flush-re-vote process makes thousands of votes just a click or two away. The only reason the Herald poll hacker got caught was his naivete and eagerness. Had he slowed down his program’s voting rate to a more naturalistic level and used IP masking, his votes would be indistinguishable from other votes. And like I said, pollsters can’t stop this from happening & it’s not technically possible. That’s why, unlike the Herald and Dominion Post, pollsters interested in real data from online polls use password controlled logins.

Even if it were possible to preclude multiple voting in online polls, they still have an enormous problems with their generalizability (the extent to which results can be assumed to be representative of a population) because of the nature of their self-selecting sampling. At best an online poll might represent the views of a website’s readership, but most websites’ poll results aren’t even representative of the site’s readership, unless all readers are equally likely to participate in the poll. In truth this isn’t so because those who do respond tend to be those who have a particular interest in the issue & ambivalent, busy or poll-savvy readers tend ignore them. In the electronic age it’s also easy for pressure groups to group-email their supporters a link to the poll, allowing mobilized interest groups to attack an online poll with just a few clicks’ effort.

So despite the fact online polls aren’t often even representative of the site readership’s views, if the results are desirable to the website’s interests they will be purported to represent “everyone”. That becomes really problematic because it assumes that “everyone” has an equal probability of participating in the online poll, even if they’ve never heard of the site, even if they’ve never heard of the issue being polled, even if they have no online access! Real polls of the entire population go to great lengths to make sure opinions are sampled from representative proportions of the demographies that comprise the entire country, in sufficient numbers to make sure margins of error are negligible. Unlike the Herald and Dominion Post, they don’t use self-selecting samples – that is, respondents aren’t those who actively seek to participate, rather they are selected by the researcher either randomly or as part of a stratified sample ensuring representative proportions of youth, elderly, North and South Islanders, Pakeha and Maori, male and female, rural and urban, etc.

Incidentally, the main problem with the accuracy of most commercial polling today, apart from priming questions, is the obsolete sampling technique of 9am to 9pm landline telephone calling, which ‘invisibilizes’ the opinions of all those without landlines (up to 67% of households in the poorest electorates), those who rely on cell phones for their principle communication channel, those who work night shifts, and those who move frequently such that their numbers are not listed – specifically, the poor and the young. Conversely, the elderly, self-employed and less transient home-owners, those who are readily contactable by such sampling methods with long-established landlines, are over-sampled. Still wondering why commercial polling tends to favour conservatives?

But back to the current fiasco of newspapers’ online polls. Doesn’t it make you wonder how many other polls might be so skewed but were never reported as such because they ‘demonstrated’ what the publisher wanted to promote? Now that the Herald and Dominion Post have acknowledged these serious weaknesses in the validity of their polls, will we see such gallant eschewal of future online poll results by their own commissioners, no matter how attractive their outcomes might be to those who published them? Will we see an improvement in the methodological integrity of polling generally? Will we see a public education campaign by news outlets on how any poll (not just the highly-vulnerable online poll) can be manipulated? I expect if the Herald or Dominion Post did a survey to see if the public had an appetite for answers to these questions, the result would be No.

Methodologically dubious polling, in terms of both priming and sampling, was epidemic in the lead up to the 2005 election and is a far greater threat to democracy than any teenage hacker able to thwart the efforts of media conglomerates like APN and Fairfax. Bogus polling is anti-democratic because it’s used to create opinions, not reflect them as they pretend to do. They create an illusion of consensus for whichever opinion is desired by the poll’s commissioning client. And when you’re talking about a public-opinion based game like politics, manufactured “public” opinion poll results can turn into self-fulfilling prophecies.

ps: The original Dominion Post story by Patrick Crewdson (published 1 Feb 2007) has since been removed from their site, so I’ve included it below. It’s a shame Parliamentary Services didn’t press the Dom Post for evidence and pursue defamation action because I know for a fact those bogus votes didn’t originate from a parliamentary server.

Parliamentary voters try to skew Key poll

Someone in Parliament now has severe finger cramps.

Yesterday’s Dominion Post poll on whether John Key will be the next prime minister proved popular with readers – particularly with those who walk the halls of power.

With more than 33,600 votes cast, 63 per cent disagreed that the National leader would succeed Helen Clark as prime minister, while 37 per cent thought he would.

But given that 17,104 of the votes were cast from parliamentary computers, Mr Key need not despair – it appears some poll participants had a vested interest.

Parliamentary workers seemingly spent yesterday supporting the status quo, 80 per cent of the votes from Parliament saying Mr Key would not be the next prime minister.

Removing all parliamentary votes from the poll meant 55 per cent favoured him as the country’s next leader.

The poll followed his first state of the nation speech.

He warned of an emerging “underclass” in New Zealand society.

Dominion Post readers were invited to vote by text message, phone, e-mail, or online at dompost.co.nz and Stuff.co.nz

Government allies outside Parliament also responded. An e-mail campaign, understood to have originated with a union, implored recipients to vote against Mr Key, urging: “go forth and vote, fellow Labourites!”

Voting in the poll was heaviest from 9am to 10am and during the lunch hour.

A related Stuff.co.nz poll had 80.9 per cent of 4000 voters supporting Mr Key’s view about a growing New Zealand “underclass”.

32 comments on “Pretend polling ”

  1. TomS 1

    The only reason you hype such un-scientific crap is to manufacture a controversy where none exists – viz, the EFB.

  2. gobsmacked 2

    Excellent article. The way these non-polls are creeping into real news coverage is quite worrying. I can’t believe any self-respecting reporter/commentator would bother referring to them, but amazingly they do.

    Online poll update:

    “Do you support eating cute baby kittens?” 97% No.

    “Do you agree with Sue Bradford, who opposes eating cute baby kittens?” 84% No.

  3. Sam Dixon 3

    is it ok to eat the ones that aren’t cute?

    because, I’m hungry

  4. Patrick 4

    Thanks for that BPGP! It really is astounding the sort of stuff these polls cover, and even more alarming, that they’re taken seriously by some quarters of our media.

  5. Santa Claws 5

    I assume this critique would also apply to pretend petitions?

  6. the sprout 6

    you of all people should know Uncky Dave.

  7. Spam 7

    I know for a fact those bogus votes didn’t originate from a parliamentary server.

    Because you know who deliberately skewed it?

  8. Santa Claws 8

    “It’s a shame Parliamentary Services didn’t press the Dom Post for evidence and pursue defamation action”

    It’s a good thing you are a pollie and not a lawyer.

    “I know for a fact those bogus votes didn’t originate from a parliamentary server.”

    Well, since you don’t provide a name, that claim is as worth as much of one of Robespierres.

  9. Lampie 9

    Dodgy polls alright, ask any statistician or marketer. TomS comment is straight to the point and so is the sproutandthebean link!

  10. Robinsod 10

    Hey DPF Claws – you don’t like me much do you? First you ban me from your blog and now you call me a liar? Poor little Claws. Oh and it should be “Robespierre’s” the apostrophe is possessive as in “the claim belonging to Robespierre”. How many times do I have to school your dumb arse in this until you get it right?!

  11. Santa Claws 11

    Well, anyone can see that its easy to Robespierres panties in a bunch.

    Call you a liar? Where’s those SIS files then? Anyway, I only said that your comments are worthless, and they are.

  12. Robinsod 12

    DPF claws – I see you’re much nicer at KB than you are here. Why don’t you try a bit harder for us mate. Oh ans the SIS file thing is a joke, y’know like so absurd as to be funny. Kinda like your politics really…

  13. Billy 13

    Robinsod, bro’, you’re not qualified to lecture on apostrophe use.

  14. Robinsod 14

    Um Billy – I am quite literally qualified to lecture on apostrophe use.

  15. hehe burnt billio. robinsod stop getting all tricky with your educashun and such like. it’s too meany on the kb trolls

  16. Billy 16

    I’m just saying that, where I come from, we put an apostrophe on the end of bro’. Old school.

  17. Robinsod 17

    Fair enough bro (just ‘cos, against all of my political instincts, I kinda like you)

  18. god, didn’t someone already try to do that one bro? i am sure i heard this nitpicky comment elsewhere….

    once “bro” has become a recognised part of the common lexicon an apostrophe is no longer justified. it is a word in its own right.

  19. Nih 19

    You’re all well behind the times brau.

  20. r0b 20

    “DPF claws – I see you’re much nicer at KB than you are here”

    Robinsod, I think we can pretty much put to rest the theory that Claws is DPF. Check out Santa’s post on KB last night (in the Phillida Bunkle thread):

    Santa Claws Says:

    November 15th, 2007 at 8:42 pm
    I can’t believe you guys I have just been reading this thread
    Shocking
    This is what the left at Kiwiblogblog has to say

    It’s precisely this thin veneer of really nasty bigotry that seemingly bubbles away under the surface of Right wing politics that keeps many centrists oriented to the Left.

    And I hate to say it but they are right The nastiest bigoted comments do in fact come from the right in the blogoshere. We are in danger of loosing our credibility. Visit public address they have less visitors and posts but the debate is really civilized and they are on the left.
    One day some enterprising Journo is going to do a story about poltics in the blogoshere,
    God lets hope they do not visit here.

    Can some one link to a Rightish site that has intelligent reasoned conversation please ?

    No way is that DPF. And as I said in another thread – Santa – good on ya. Well done for speaking up on KB.

    Can anyone help Santa in his search for a civil right wing blog? Santa, if you find one, will you let us know here?

  21. Spam 21

    AFAIK, every blog DPF posts on, he does it under his own name / moniker.

  22. Billy 22

    I have to say that, of late, I prefer coming here as well. For one thing, what’s the point of talking to a bunch of people who agree with you? And for another, it’s not a good thing having D4J on your side. But in case you lefties get complacent, remember, Maia is on your side.

  23. r0b 23

    “I have to say that, of late, I prefer coming here as well. For one thing, what’s the point of talking to a bunch of people who agree with you?”

    Billy – there’s a good range of opinion here. And despite the fact that there are many many lapses, the tone is still mostly far better here than at KB.

    Now, I may regret asking this, but who is Maia?

  24. i have played a large part in destroying the tone of the standard today- and for that i apologise

    normally i am quite sane billy. do come and join us more often!

    maia has some good stuff to say though…

  25. Billy 25

    r0b, there’s about the same range as at KB.

    Maia is a complete piece of work. My favourite:
    http://www.imdb.com/gallery/mptv/1091/0959_2116.jpg.html?seq=14

    She is quite serious, apparently. Enjoy.

  26. r0b 26

    “i have played a large part in destroying the tone of the standard today- and for that i apologise”

    What? Bean – don’t fret – I’m a big fan of fun and hijinks! Love your work!

    It’s mindless personal attacks and ranting that bring blogs down, and these are what I hope The Standard can minimise. If this is a place where dissenting views are treated with respect (and then debated to bits!) then I’ll be happy.

  27. r0b 27

    Billy – huh? – sorry if I’m dim, but I don’t get the link?

  28. Billy 28

    Only because I posted the wrong one. Take a look at this and tell me this woman is not as mad as a cut snake:

    http://capitalismbad.blogspot.com/2006/07/beautiful-boy.html

  29. Nih 29

    She seems to have some disorders. Good thing I’m such a big fan of crazy people using the internet.

  30. AncientGeek 30

    Getting back to the topic. That is a good article.

    Online polls are a waste of time – at best they are a media space filler, and at worst they’re used to try and manipulate public opinion by editorial comment. As a programmer, I find them ridiculous – it takes minutes to manipulate them. They are so ridiculously easy I’ve never bothered to write code to do one myself – but I have given advice on how to do it. I helped advise on that DomPost one – and whoever wrote that civil servants did it is either lying or been lied to.

    It isn’t even illegal to ummm adjust the online polls. It isn’t criminal to use whatever ‘browser’ you want on a public website, and to fill in fields with whatever means you want. Sending e-mails to an e-mail poll – well the best that you could get done with is for spam – but they requested e-mail. There is no contract to enforce – where is the consideration. I’m sure that the police could figure out a creative charge (seem to do that a lot recently), but it would be thrown out of court eventually.

    However I suspect that online polls are also ineffective. The only time I’ve ever heard anyone talking about them has been when I’m associating with the chattering classes. The politicians, journo’s, activists, and of course the online equivalent of talkback radio – blogs. I suspect you probably also hear about them on real talkback – but I avoid that.

    I don’t hear about them at work, in the extended family, friends, aqquantince, etc. They aren’t raised when I’m vigorously discussing politics or the economy. ‘Normal’ people have enough sense to know that you can’t trust online polls – they don’t even trust the ‘scientific’ polls.

    I only hear about them when there is some idiot with an ego wanting push their point of view. They are the refuge of the feeble-minded – I can’t get people to agree with me, so I’ll pretend that they do.

    Pretty much what the person in the article said.

  31. the sprout 31

    nicely said AG

Links to post

Recent Comments

Recent Posts

  • What’s Labour achieved so far?
    Quite a bit! This Government was elected to take on the toughest issues facing Aotearoa – and that’s what we’re doing. Since the start of the pandemic, protecting lives and livelihoods has been a priority, but we’ve also made progress on long-term challenges, to deliver a future the next generation ...
    6 days ago
  • Tackling the big issues in 2022
    This year, keeping Kiwis safe from COVID will remain a key priority of the Government – but we’re also pushing ahead on some of New Zealand’s biggest long-term challenges. In 2022, we’re working to get more Kiwis into homes, reduce emissions, lift children out of poverty, and ensure people get ...
    1 week ago
  • Happy new year, Aotearoa!
    Welcome to 2022! As we look ahead to another year of progress on the big issues facing our country, we’re taking a look back at the year that’s been and everything the team of five million achieved together in 2021. ...
    3 weeks ago

  • New Zealand prepared to send support to Tonga
    New Zealand is ready to assist Tonga in its recovery from Saturday night’s undersea eruption and tsunami, Foreign Affairs Minister Nanaia Mahuta and Defence Minister Peeni Henare said today. “Following the successful surveillance and reconnaissance flight of a New Zealand P-3K2 Orion on Monday, imagery and details have been sent ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Aotearoa New Zealand stands ready to assist people of Tonga
    The thoughts of New Zealanders are with the people of Tonga following yesterday’s undersea volcanic eruption and subsequent tsunami waves, Foreign Affairs Minister Nanaia Mahuta says. “Damage assessments are under way and New Zealand has formally offered to provide assistance to Tonga,” said Nanaia Mahuta. New Zealand has made an ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Record high of new homes consented continues
    In the year ended November 2021, 48,522 new homes were consented, up 26 per cent from the November 2020 year. In November 2021, 4,688 new dwellings were consented. Auckland’s new homes consented numbers rose 25 per cent in the last year. Annual figures for the last nine months show more ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Report trumpets scope for ice cream exports
    Latest research into our premium ice cream industry suggests exporters could find new buyers in valuable overseas markets as consumers increasingly look for tip top quality in food. Economic Development Minister Stuart Nash has released a new report for the Food and Beverage Information Project. The project is run by ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Honouring the legacy of legendary kaumātua Muriwai Ihakara
    Associate Minister for Arts, Culture, and Heritage Kiri Allan expressed her great sadness and deepest condolences at the passing of esteemed kaumātua, Muriwai Ihakara. “Muriwai’s passing is not only a loss for the wider creative sector but for all of Aotearoa New Zealand. The country has lost a much beloved ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Have your say on proposed changes to make drinking water safer
    Associate Minister for the Environment Kiri Allan is urging all New Zealanders to give feedback on proposed changes aimed at making drinking water safer. “The current regulations are not fit for purpose and don’t offer enough protection, particularly for those whose water comes from smaller supplies,” Kiri Allan said. “This ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Planting the seeds for rewarding careers
    A boost in funding for a number of Jobs for Nature initiatives across Canterbury will provide sustainable employment opportunities for more than 70 people, Conservation Minister Kiri Allan says. “The six projects are diverse, ranging from establishing coastline trapping in Kaikōura, to setting up a native plant nursery, restoration planting ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • New Zealand congratulates Tonga's new Prime Minister on appointment
    Minister of Foreign Affairs Nanaia Mahuta today congratulated Hon Hu'akavameiliku Siaosi Sovaleni on being appointed Prime Minister of the Kingdom of Tonga. “Aotearoa New Zealand and Tonga have an enduring bond and the Kingdom is one of our closest neighbours in the Pacific. We look forward to working with Prime ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • High-tech investment extends drought forecasting for farmers and growers
    The Government is investing in the development of a new forecasting tool that makes full use of innovative climate modelling to help farmers and growers prepare for dry conditions, Agriculture Minister Damien O'Connor said.  The new approach, which will cost $200,000 and is being jointly funded through the Ministry for ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Support for fire-hit Waiharara community
    The government will contribute $20,000 towards a Mayoral Relief Fund to support those most affected by the fires in Waiharara in the Far North, Minister for Emergency Management Kiri Allan says. “I have spoken to Far North Mayor John Carter about the effect the fires continue to have, on residents ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Manawatū’s ‘oases of nature’ receive conservation boost
    The Government is throwing its support behind projects aimed at restoring a cluster of eco-islands and habitats in the Manawatū which were once home to kiwi and whio. “The projects, which stretch from the Ruahine Ranges to the Horowhenua coastline, will build on conservation efforts already underway and contribute ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • New Zealand to continue Solomon Islands support
    A New Zealand Defence Force and Police deployment to help restore peace and stability to Solomon Islands is being scaled down and extended. The initial deployment followed a request for support from Solomon Islands Government after riots and looting in capital Honiara late last month. They joined personnel from Australia, ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Our Pacific community shares in New Year’s Honours
    Prominent Pacific health champion Faumuina Professor Fa’afetai Sopoaga has been made a Companion of the New Zealand Order of Merit in the New Year’s Honours list. Professor Sopoaga has been a champion for Pacific Health at Otago University, said Minister of Pacific Peoples Aupito William Sio. “She’s overseen changes in ...
    BeehiveBy beehive.govt.nz
    3 weeks ago
  • Congratulations to Māori New Year’s Honours stars of 2022
    Kei aku rangatira kua whakawhiwhia koutou ki ngā tohu ā tō tātou kuīni hei whakanui nui i ā koutou mahi rangatira i hāpai i te manotini puta noa i a Aotearoa. Ko koutou ngā tino tauira. I whanake i ngā hapori, iwi, hapū, whānau me te motu anō hoki. Mauri ...
    BeehiveBy beehive.govt.nz
    3 weeks ago
  • Top honours for women in sport
    Minister of Sport and Recreation Grant Robertson has congratulated Olympian Lisa Carrington and Paralympian Sophie Pascoe on being made Dames Companion of the New Zealand Order of Merit (DNZM) in the 2022 New Year Honours. Lisa Carrington is New Zealand’s most successful Olympian, having won five gold and one bronze ...
    BeehiveBy beehive.govt.nz
    3 weeks ago
  • PM congratulates 2022 New Year Honours recipients
    The New Zealanders recognised in the New Year 2022 Honours List represent the determination and service exemplified by so many New Zealanders during what has been another tough year due to the global COVID-19 pandemic, Prime Minister Jacinda Ardern said. “I never fail to be amazed by the outstanding things ...
    BeehiveBy beehive.govt.nz
    3 weeks ago