Pretend polling

Written By: - Date published: 10:03 am, November 16th, 2007 - 32 comments
Categories: polls - Tags:

Dodgy online polls seem to be becoming a staple of political reporting here in New Zealand. The story earlier this week on poll “hacking” sent me in search of an insightful opinion I’d heard previously.

So today, we have a guest author on The Standard – BPGP, on “Pretend polling”.

Pretend Polling
BPGP

So for the second time this year a newspaper publisher has made a story out of the fictions that are their online poll results. The first was a piece earlier this year in the Dominion Post alleging Parliamentary staff had skewed their online preferred PM poll to show Clark was ahead of Key. And now we have the Herald shrieking about how a teenaged hacker skewed their online poll to show that readers didn’t think New Zealand was “becoming a less free and democratic country”, despite the Herald’s concerted efforts to convince its readers we’re heading for a Stalinist dystopia.

Both these stories demonstrate just how bogus online polls really are, and it makes me doubt their creators have ever heard of the terms “validity” or “sampling”. It is interesting though, that despite most online newspaper polls being regularly quoted as if they were the gospel truth (at least when their outcomes show opinions advantageous to their publishers’ editorial interests), these two polls are miraculously exposed as skewed.

Sometimes publishers are generous enough to mention that the polls “aren’t scientific” & a discrete way of saying they have no validity. But invariably the next sentence will go on to quote the results as if they do have meaning that can be generalized to the greater population. After all, why would you bother to run a poll if it was actually completely meaningless?

Well, there are several reasons, but few of them to do with illuminating public opinion.

One is that online polls are useful for giving the impression that publishers interact with their audiences and are genuinely interested in listening. You might say that polls help media outlets to gauge the opinion of their audiences, but in reality they aren’t so foolish as to alter their editorial stances on the basis of such flawed data (papers like the Herald use a more sound, random sampling procedure to invite clickers to participate in a Readers’ Survey for that). Online polls are also a useful gimmick for keeping up online readers’ interest and so contributing to the maintenance of hit rates.

But the main reason for using online polling is to create an impression of popular support for the outlet’s editorial stance. If you were really cynical you might think that poll commissioners just pump the data in whichever way they want it, but in reality it’s quite unnecessary to engage in such risky manipulation. There are other ways. Like priming readers with skewed reporting on the issue, such that so long as they rely on that outlet for their information on a particular issue, they are bound to come to similar conclusions as those promoted by the outlet’s reporting.

Another way is to use priming questions. The Section 59 “debate” was a gold plated example, asking questions like “Should parents have the right to discipline their children?” when in fact the bill had nothing to do with “rights to discipline”. And then there’s misrepresenting what the poll question actually meant, as in saying that because a majority understandably agree that “parents have the right to discipline their children”, they therefore are opposed to Section 59. Yet another way is to massage the response options. Typically, very complex questions are forced into Yes/No dichotomies, when respondents may in fact have more moderate or complicated responses.

Similarly, you can alter a result just by the number of response options you allow. For example, you could ask “Do you support the EFB – Yes or No?” and get 60% for and 40% against. But if you ask “Do you support the EFB – Yes, No, or Yes but with some changes?” you might get 30% Yes, 40% No, and 30% Yes but with some changes. These results of course, would then be reported as “70% percent opposed to EFB in current form”, when the same question with just a Yes/No format would have provided an unhelpful result (for the Herald at least) showing majority support for the EFB.

The Herald and Dominion Post articles point to another, user driven form of manipulating the results – multiple voting. No matter what newspapers claim about security to measures to stop such behaviour, they will always be vulnerable to it. Cookies and IP tracking to monitor voting behaviour can easily be circumvented with SSL-flushing between votes and freeware IP masking software. A little bit of script to automate the vote-flush-re-vote process makes thousands of votes just a click or two away. The only reason the Herald poll hacker got caught was his naivete and eagerness. Had he slowed down his program’s voting rate to a more naturalistic level and used IP masking, his votes would be indistinguishable from other votes. And like I said, pollsters can’t stop this from happening & it’s not technically possible. That’s why, unlike the Herald and Dominion Post, pollsters interested in real data from online polls use password controlled logins.

Even if it were possible to preclude multiple voting in online polls, they still have an enormous problems with their generalizability (the extent to which results can be assumed to be representative of a population) because of the nature of their self-selecting sampling. At best an online poll might represent the views of a website’s readership, but most websites’ poll results aren’t even representative of the site’s readership, unless all readers are equally likely to participate in the poll. In truth this isn’t so because those who do respond tend to be those who have a particular interest in the issue & ambivalent, busy or poll-savvy readers tend ignore them. In the electronic age it’s also easy for pressure groups to group-email their supporters a link to the poll, allowing mobilized interest groups to attack an online poll with just a few clicks’ effort.

So despite the fact online polls aren’t often even representative of the site readership’s views, if the results are desirable to the website’s interests they will be purported to represent “everyone”. That becomes really problematic because it assumes that “everyone” has an equal probability of participating in the online poll, even if they’ve never heard of the site, even if they’ve never heard of the issue being polled, even if they have no online access! Real polls of the entire population go to great lengths to make sure opinions are sampled from representative proportions of the demographies that comprise the entire country, in sufficient numbers to make sure margins of error are negligible. Unlike the Herald and Dominion Post, they don’t use self-selecting samples – that is, respondents aren’t those who actively seek to participate, rather they are selected by the researcher either randomly or as part of a stratified sample ensuring representative proportions of youth, elderly, North and South Islanders, Pakeha and Maori, male and female, rural and urban, etc.

Incidentally, the main problem with the accuracy of most commercial polling today, apart from priming questions, is the obsolete sampling technique of 9am to 9pm landline telephone calling, which ‘invisibilizes’ the opinions of all those without landlines (up to 67% of households in the poorest electorates), those who rely on cell phones for their principle communication channel, those who work night shifts, and those who move frequently such that their numbers are not listed – specifically, the poor and the young. Conversely, the elderly, self-employed and less transient home-owners, those who are readily contactable by such sampling methods with long-established landlines, are over-sampled. Still wondering why commercial polling tends to favour conservatives?

But back to the current fiasco of newspapers’ online polls. Doesn’t it make you wonder how many other polls might be so skewed but were never reported as such because they ‘demonstrated’ what the publisher wanted to promote? Now that the Herald and Dominion Post have acknowledged these serious weaknesses in the validity of their polls, will we see such gallant eschewal of future online poll results by their own commissioners, no matter how attractive their outcomes might be to those who published them? Will we see an improvement in the methodological integrity of polling generally? Will we see a public education campaign by news outlets on how any poll (not just the highly-vulnerable online poll) can be manipulated? I expect if the Herald or Dominion Post did a survey to see if the public had an appetite for answers to these questions, the result would be No.

Methodologically dubious polling, in terms of both priming and sampling, was epidemic in the lead up to the 2005 election and is a far greater threat to democracy than any teenage hacker able to thwart the efforts of media conglomerates like APN and Fairfax. Bogus polling is anti-democratic because it’s used to create opinions, not reflect them as they pretend to do. They create an illusion of consensus for whichever opinion is desired by the poll’s commissioning client. And when you’re talking about a public-opinion based game like politics, manufactured “public” opinion poll results can turn into self-fulfilling prophecies.

ps: The original Dominion Post story by Patrick Crewdson (published 1 Feb 2007) has since been removed from their site, so I’ve included it below. It’s a shame Parliamentary Services didn’t press the Dom Post for evidence and pursue defamation action because I know for a fact those bogus votes didn’t originate from a parliamentary server.

Parliamentary voters try to skew Key poll

Someone in Parliament now has severe finger cramps.

Yesterday’s Dominion Post poll on whether John Key will be the next prime minister proved popular with readers – particularly with those who walk the halls of power.

With more than 33,600 votes cast, 63 per cent disagreed that the National leader would succeed Helen Clark as prime minister, while 37 per cent thought he would.

But given that 17,104 of the votes were cast from parliamentary computers, Mr Key need not despair – it appears some poll participants had a vested interest.

Parliamentary workers seemingly spent yesterday supporting the status quo, 80 per cent of the votes from Parliament saying Mr Key would not be the next prime minister.

Removing all parliamentary votes from the poll meant 55 per cent favoured him as the country’s next leader.

The poll followed his first state of the nation speech.

He warned of an emerging “underclass” in New Zealand society.

Dominion Post readers were invited to vote by text message, phone, e-mail, or online at dompost.co.nz and Stuff.co.nz

Government allies outside Parliament also responded. An e-mail campaign, understood to have originated with a union, implored recipients to vote against Mr Key, urging: “go forth and vote, fellow Labourites!”

Voting in the poll was heaviest from 9am to 10am and during the lunch hour.

A related Stuff.co.nz poll had 80.9 per cent of 4000 voters supporting Mr Key’s view about a growing New Zealand “underclass”.

32 comments on “Pretend polling ”

  • Comments are now closed

  1. TomS 1
    16 November 2007 at 10:24 am

    The only reason you hype such un-scientific crap is to manufacture a controversy where none exists – viz, the EFB.

  2. gobsmacked 2
    16 November 2007 at 10:35 am

    Excellent article. The way these non-polls are creeping into real news coverage is quite worrying. I can’t believe any self-respecting reporter/commentator would bother referring to them, but amazingly they do.

    Online poll update:

    “Do you support eating cute baby kittens?” 97% No.

    “Do you agree with Sue Bradford, who opposes eating cute baby kittens?” 84% No.

  3. Sam Dixon 3
    16 November 2007 at 10:48 am

    is it ok to eat the ones that aren’t cute?

    because, I’m hungry

  4. Patrick 4
    16 November 2007 at 10:54 am

    Thanks for that BPGP! It really is astounding the sort of stuff these polls cover, and even more alarming, that they’re taken seriously by some quarters of our media.

  5. Santa Claws 5
    16 November 2007 at 12:00 pm

    I assume this critique would also apply to pretend petitions?

  6. the sprout 6
    16 November 2007 at 12:22 pm

    you of all people should know Uncky Dave.

  7. Spam 7
    16 November 2007 at 1:02 pm

    I know for a fact those bogus votes didn’t originate from a parliamentary server.

    Because you know who deliberately skewed it?

  8. Santa Claws 8
    16 November 2007 at 1:56 pm

    “It’s a shame Parliamentary Services didn’t press the Dom Post for evidence and pursue defamation action”

    It’s a good thing you are a pollie and not a lawyer.

    “I know for a fact those bogus votes didn’t originate from a parliamentary server.”

    Well, since you don’t provide a name, that claim is as worth as much of one of Robespierres.

  9. Lampie 9
    16 November 2007 at 2:16 pm

    Dodgy polls alright, ask any statistician or marketer. TomS comment is straight to the point and so is the sproutandthebean link!

  10. Robinsod 10
    16 November 2007 at 2:28 pm

    Hey DPF Claws – you don’t like me much do you? First you ban me from your blog and now you call me a liar? Poor little Claws. Oh and it should be “Robespierre’s” the apostrophe is possessive as in “the claim belonging to Robespierre”. How many times do I have to school your dumb arse in this until you get it right?!

  11. Santa Claws 11
    16 November 2007 at 3:28 pm

    Well, anyone can see that its easy to Robespierres panties in a bunch.

    Call you a liar? Where’s those SIS files then? Anyway, I only said that your comments are worthless, and they are.

  12. Robinsod 12
    16 November 2007 at 3:45 pm

    DPF claws – I see you’re much nicer at KB than you are here. Why don’t you try a bit harder for us mate. Oh ans the SIS file thing is a joke, y’know like so absurd as to be funny. Kinda like your politics really…

  13. Billy 13
    16 November 2007 at 3:48 pm

    Robinsod, bro’, you’re not qualified to lecture on apostrophe use.

  14. Robinsod 14
    16 November 2007 at 3:50 pm

    Um Billy – I am quite literally qualified to lecture on apostrophe use.

  15. the bean 15
    16 November 2007 at 4:09 pm

    hehe burnt billio. robinsod stop getting all tricky with your educashun and such like. it’s too meany on the kb trolls

  16. Billy 16
    16 November 2007 at 4:12 pm

    I’m just saying that, where I come from, we put an apostrophe on the end of bro’. Old school.

  17. Robinsod 17
    16 November 2007 at 4:20 pm

    Fair enough bro‘ (just ‘cos, against all of my political instincts, I kinda like you)

  18. the bean 18
    16 November 2007 at 4:25 pm

    god, didn’t someone already try to do that one bro? i am sure i heard this nitpicky comment elsewhere….

    once “bro” has become a recognised part of the common lexicon an apostrophe is no longer justified. it is a word in its own right.

  19. Nih 19
    16 November 2007 at 4:27 pm

    You’re all well behind the times brau.

  20. r0b 20
    16 November 2007 at 4:49 pm

    “DPF claws – I see you’re much nicer at KB than you are here”

    Robinsod, I think we can pretty much put to rest the theory that Claws is DPF. Check out Santa’s post on KB last night (in the Phillida Bunkle thread):

    Santa Claws Says:

    November 15th, 2007 at 8:42 pm
    I can’t believe you guys I have just been reading this thread
    Shocking
    This is what the left at Kiwiblogblog has to say

    It’s precisely this thin veneer of really nasty bigotry that seemingly bubbles away under the surface of Right wing politics that keeps many centrists oriented to the Left.

    And I hate to say it but they are right The nastiest bigoted comments do in fact come from the right in the blogoshere. We are in danger of loosing our credibility. Visit public address they have less visitors and posts but the debate is really civilized and they are on the left.
    One day some enterprising Journo is going to do a story about poltics in the blogoshere,
    God lets hope they do not visit here.

    Can some one link to a Rightish site that has intelligent reasoned conversation please ?

    No way is that DPF. And as I said in another thread – Santa – good on ya. Well done for speaking up on KB.

    Can anyone help Santa in his search for a civil right wing blog? Santa, if you find one, will you let us know here?

  21. Spam 21
    16 November 2007 at 5:26 pm

    AFAIK, every blog DPF posts on, he does it under his own name / moniker.

  22. Billy 22
    16 November 2007 at 5:33 pm

    I have to say that, of late, I prefer coming here as well. For one thing, what’s the point of talking to a bunch of people who agree with you? And for another, it’s not a good thing having D4J on your side. But in case you lefties get complacent, remember, Maia is on your side.

  23. r0b 23
    16 November 2007 at 5:36 pm

    “I have to say that, of late, I prefer coming here as well. For one thing, what’s the point of talking to a bunch of people who agree with you?”

    Billy – there’s a good range of opinion here. And despite the fact that there are many many lapses, the tone is still mostly far better here than at KB.

    Now, I may regret asking this, but who is Maia?

  24. the bean 24
    16 November 2007 at 5:41 pm

    i have played a large part in destroying the tone of the standard today- and for that i apologise

    normally i am quite sane billy. do come and join us more often!

    maia has some good stuff to say though…

  25. Billy 25
    16 November 2007 at 5:44 pm

    r0b, there’s about the same range as at KB.

    Maia is a complete piece of work. My favourite:
    http://www.imdb.com/gallery/mptv/1091/0959_2116.jpg.html?seq=14

    She is quite serious, apparently. Enjoy.

  26. r0b 26
    16 November 2007 at 5:52 pm

    “i have played a large part in destroying the tone of the standard today- and for that i apologise”

    What? Bean – don’t fret – I’m a big fan of fun and hijinks! Love your work!

    It’s mindless personal attacks and ranting that bring blogs down, and these are what I hope The Standard can minimise. If this is a place where dissenting views are treated with respect (and then debated to bits!) then I’ll be happy.

  27. r0b 27
    16 November 2007 at 5:55 pm

    Billy – huh? – sorry if I’m dim, but I don’t get the link?

  28. Billy 28
    16 November 2007 at 7:06 pm

    Only because I posted the wrong one. Take a look at this and tell me this woman is not as mad as a cut snake:

    http://capitalismbad.blogspot.com/2006/07/beautiful-boy.html

  29. Nih 29
    16 November 2007 at 7:09 pm

    She seems to have some disorders. Good thing I’m such a big fan of crazy people using the internet.

  30. AncientGeek 30
    17 November 2007 at 11:41 am

    Getting back to the topic. That is a good article.

    Online polls are a waste of time – at best they are a media space filler, and at worst they’re used to try and manipulate public opinion by editorial comment. As a programmer, I find them ridiculous – it takes minutes to manipulate them. They are so ridiculously easy I’ve never bothered to write code to do one myself – but I have given advice on how to do it. I helped advise on that DomPost one – and whoever wrote that civil servants did it is either lying or been lied to.

    It isn’t even illegal to ummm adjust the online polls. It isn’t criminal to use whatever ‘browser’ you want on a public website, and to fill in fields with whatever means you want. Sending e-mails to an e-mail poll – well the best that you could get done with is for spam – but they requested e-mail. There is no contract to enforce – where is the consideration. I’m sure that the police could figure out a creative charge (seem to do that a lot recently), but it would be thrown out of court eventually.

    However I suspect that online polls are also ineffective. The only time I’ve ever heard anyone talking about them has been when I’m associating with the chattering classes. The politicians, journo’s, activists, and of course the online equivalent of talkback radio – blogs. I suspect you probably also hear about them on real talkback – but I avoid that.

    I don’t hear about them at work, in the extended family, friends, aqquantince, etc. They aren’t raised when I’m vigorously discussing politics or the economy. ‘Normal’ people have enough sense to know that you can’t trust online polls – they don’t even trust the ‘scientific’ polls.

    I only hear about them when there is some idiot with an ego wanting push their point of view. They are the refuge of the feeble-minded – I can’t get people to agree with me, so I’ll pretend that they do.

    Pretty much what the person in the article said.

  31. the sprout 31
    17 November 2007 at 11:57 am

    nicely said AG

Links to post

The server will be getting hardware changes this evening starting at 10pm NZDT.
The site will be off line for some hours.