Here's how to make opinion polls more representative and honest

In 2012, US statistician Nate Silver correctly predicted the results of all 50 states in the US presidential election with 100% accuracy.

Compare this with the shock result of the 2016 Trump election, and now the 2019 Australian election, when nearly all Australian opinion polls incorrectly predicted that Bill Shorten’s Labor Party would defeat Scott Morrison’s Coalition on a two-party preferred basis.

How did the pollsters get it so wrong? More importantly, how can we ensure we get a more accurate result next time?

Read more: State of the states: Queensland and Tasmania win it for the Coalition

What opinion polls must do

A good opinion poll must do two difficult things:

1. it must find a large and representative sample of Australian voters
2. it must get those voters to honestly divulge who they will vote for.

If both of these things happen, statistical theory gives us confidence that the poll will be reliable.

To add to the difficulty, we don’t just need good opinion polls we need excellent opinion polls, because even a small error could mean predicting the wrong winner.

For example, if a government ministry wanted to know what percentage of people had done volunteer work in the last year, it probably wouldn’t matter for their purposes if they commissioned two polls which gave estimates of 48% and 52%. But this variation is unacceptable for electoral opinion polls, which need a small margin of error to accurately predict the winner.

How to get a representative sample

A good representative sample will be the whole Australian voting population, but in microcosm. If there are 16 million eligible voters, then a representative sample of 1,600 would mean that each sampled voter represents 10,000 people just like them.

Using relatively small samples is a cheap way to estimate the bigger picture of the election, without resorting to expensive surveys like the Australian same-sex marriage survey.

With a large and representative sample, statistical theory tells us that we will get close to the real result with a narrow margin of error. But the opinion polls in this election consistently predicted the wrong winner, meaning there was something wrong with the samples.

Read more: Coalition wins election but Abbott loses Warringah, plus how the polls got it so wrong

The problem of disruptive technology

Opinion polls are usually implemented by pollsters calling landlines, sometimes calling mobile phones, robo-dialling and internet surveys.

In the recent past when almost every Australian had a landline phone, it was far easier to get a representative sample of voters by randomly calling numbers or randomly sampling from the White Pages. The randomness is key, because it destroys bias by giving every eligible voter the same chance of being surveyed. This in turn avoids favouring particular groups.

Modern methods of communication have spoiled the representativeness of landlines – 36% of Australian adults now use a mobile only. And because older Australians are more likely to keep their landline, this biases any sample that uses landlines alone.

Far more Australians have a mobile phone than a landline – so why not use these instead? One reason is that some people have two mobiles – business and personal – so the poll would over-represent working Australians. And with fewer older people having mobiles, this approach would also over-represent younger Australians.

Even if we had a perfectly representative sample of phone numbers, we can expect a large number of people not to answer or hang up, which creates another bias because the poll then over-represents people with more time and those who are more politically engaged.

How to get honest answers

Opinion polls have been wrong before. Two examples are the 1992 and 2015 elections in the UK, where the polls wrongly predicted a good result for Labour. The post-mortem of these failings came up with the theory of the “Shy Tory factor”, where people were too embarrassed to admit to pollsters that they were voting for the Tories (the Conservative right-wing party).

Plenty of Australians are not embarrassed to tell anyone who they will vote for, but even the proportion of shy voters is small, then they could still derail the accuracy of the poll.

A number of Australian voters have admitted to giving fake answers to telephone polls because they were annoyed by the call. Even a small percent of annoyed people could ruin an otherwise well-designed opinion poll.

Another theory – put forward by Australian Nobel Laureate Brian Schmidt – is that the pollsters listened to each other more than they did the public. This is known as “herding”, where poll results that seemed to far from the norm were shepherded towards a groupthink.

An analysis of the numbers does show a striking lack of variability in the many recent polls.

How to get better polls

Better opinions polls are more expensive because pollsters need to spend more effort getting a representative and honest sample of voters. Some of the quick and easy polls that are currently used are primarily aimed at generating news stories, with accuracy as a secondary goal.

Poll accuracy could be increased by paying people to participate, and paying them more if, as a group, they get the result right. This is an untested idea, but it could create a strong incentive for people share their true intentions.

The same argument could also improve the pollsters’ strategies. That is, create financial rewards for getting things right, or penalties for getting them wrong. However, with shrinking budgets at news outlets to finance polling, there is no reason to believe that this problem is going to fix itself.

Read more: Going up. Monday showed what the market thinks of Morrison

Polling is a difficult job

An opinion poll targets the shifting opinions of an ever-changing population. Voters are often difficult to reach or recalcitrant. Yet much of the population is likely unhappy with the polls. They may never pay heed to polls again, or refuse to participate in them, which will ironically make the pollster’s job even harder.

Election opinion polls are like penalty shoot-outs at a World Cup final: there’s huge pressure to get it right and we remember the big misses most of all. But come the next World Cup we’re back watching the penalties, because nobody has come up with a better system to assess the national mood.

Adrian Barnett receives funding from The National Health and Medical Research Council.

Scott Sisson is Professor, Future Fellow and Head of Statistics at UNSW. He is a past-President of the Statistical Society of Australia and a Deputy Director of the Australian Centre of Excellence in Mathematical and Statistical Frontiers. He is a member of the Methodology Advisory Committee of the Australian Bureau of Statistics.