Cliff Zukin writes in the NY Times that those paying close attention to the 2016 election should exercise caution as they read the polls because election polling is in near crisis as statisticians say polls are becoming less reliable. According to Zukin, two trends are driving the increasing unreliability of election and other polling in the United States: the growth of cellphones and the decline in people willing to answer surveys. Coupled, they have made high-quality research much more expensive to do, so there is less of it. This has opened the door for less scientifically based, less well-tested techniques. To top it off, a perennial election polling problem, how to identify “likely voters,” has become even thornier. Today, a majority of people are difficult or impossible to reach on landline phones. One problem is that the 1991 Telephone Consumer Protection Act has been interpreted by the Federal Communications Commission to prohibit the calling of cellphones through automatic dialers, in which calls are passed to live interviewers only after a person picks up the phone. To complete a 1,000-person survey, it’s not unusual to have to dial more than 20,000 random numbers, most of which do not go to actual working telephone numbers.
The second unsettling trend are rapidly declining response rates, reaching levels once considered unimaginable. In the late 1970s, pollsters considered an 80 percent response rate acceptable but by 2014 the response rate has fallen to 8 percent. "Our old paradigm has broken down, and we haven’t figured out how to replace it," concludes Zukin. "In short, polls and pollsters are going to be less reliable. We may not even know when we’re off base. What this means for 2016 is anybody’s guess."
Original Submission
Related Stories
A survey by FiveThirtyEight has found that pollsters believe their public reputations are declining:
No votes have been cast yet in the 2016 election, but there may already be one set of losers in the campaign: pollsters' reputations. And that's according to the pollsters themselves.
We asked people working at some of the nation's most prominent polling outfits whether pollsters' public image has improved or declined since the 2012 election. Of the 21 who answered, none said their public image had improved, and two-thirds said it had declined.
That was one of a few dozen questions we posed to 76 of the most prolific and prominent political pollsters. And we found that the people who are measuring and shaping public perception of the election — as well as Donald Trump's Twitter account — are feeling much more positive about the work they do than they think everyone else does. (You can find the questionnaire in this PDF, all the responses on GitHub, and a list of the pollsters who responded in the footnotes.)
Using a classic tactic in politics, many pollsters blamed the media. There were three strands to their criticism. The first is that the media make too much of bad moments for the industry, when its polls miss election results badly. (We've covered those moments in last year's midterms, as well as in Israel, the U.K., Greece and Kentucky.) The second is that media organizations that aggregate polls combine the bad with the good, tarnishing all for the sins of a few. (Four pollsters said aggregators are doing badly or very badly at filtering out polls from bad polling organizations, four said they were doing OK and four said they were doing well.) And the third is that uncritical media reports of outlandish claims — such as Ben Carson's that Egypt's pyramids were used to store grain — leads many Americans to believe outlandish things, which pollsters are blamed for quantifying.
"Polls are wrong is a more interesting story than when the polls do well," said Barbara Carvalho of Marist College. "Lumping all methods together distorts the accuracy of polling."
But several respondents acknowledged that the media wouldn't have a story if pollsters were nailing election results. "Obviously, there were several high-profile calamities in the past three years," said Matthew Towery of Opinion Savvy. "The best I can say is this: The field is evolving, and some pollsters are succumbing to natural selection."
Related: Political Polls Become Less Reliable As We Head into 2016 Presidential Election
(Score: 1, Disagree) by Anonymous Coward on Monday June 22 2015, @08:12PM
Asking a few thousand people a few questions and then assuming the answers were honest and accurate isn't exactly scientific, after all.
(Score: 0, Disagree) by Anonymous Coward on Monday June 22 2015, @08:36PM
Disagree?
Want to post a response to that?
I thought it was a very valid point although it is better to make that statement and back it up with some data.
(Score: 3, Insightful) by Tork on Monday June 22 2015, @09:18PM
I thought it was a very valid point...
There's your problem.
🏳️🌈 Proud Ally 🏳️🌈 - Give us ribbiti or make us croak! 🐸
(Score: 0) by Anonymous Coward on Tuesday June 23 2015, @02:23AM
Yeah it is a problem to use things like reasoning and logic when discussing science. It is like three quarters of the people here take it as a religion and it is blasphemy to point out when something actually is not scientific at all.
(Score: 3, Touché) by Tork on Tuesday June 23 2015, @05:02AM
Yeah it is a problem to use things like reasoning and logic when discussing science.
Try it.
🏳️🌈 Proud Ally 🏳️🌈 - Give us ribbiti or make us croak! 🐸
(Score: 0) by Anonymous Coward on Tuesday June 23 2015, @02:06PM
You try, since all you've been doing is making very short comments that explain absolutely nothing. How insightful.
(Score: 2) by Tork on Tuesday June 23 2015, @03:05PM
🏳️🌈 Proud Ally 🏳️🌈 - Give us ribbiti or make us croak! 🐸
(Score: 3, Insightful) by tibman on Monday June 22 2015, @09:38PM
Not sure why people are hating on you. Land-line based phones would be a terrible way to get a cross-section of voter's intentions. It was probably useful ten years ago but not now.
SN won't survive on lurkers alone. Write comments.
(Score: 2) by aristarchus on Tuesday June 23 2015, @08:50AM
Not sure why people are hating on you.
Not hating, just pointing out an idiot when one raises its ugly head. If the so egregoiusly down-modded poster had the slighted understanding of sampling, statistics, inductive reasoning, and just plain SCIENCE, they would not have said such a stupid thing. And now you, you have stepped right in with your lack of understanding of what is being discussed. The poster being so properly chastised said nothing about landlines, that is actually something that is in the FA! And now you defend them by making it seem like the post may have been on topic? Fey, fey, my dear sir (or madam), these are dangerous times, when we must choose our terms carefully, lest they become fodder for such idiots as above.
Now about sampling (and I am quite sure that professional pollsters are well aware of their difficulties), everyone should be aware of the US presidential election of 1948, where the Chicago https://en.wikipedia.org/wiki/Dewey_Defeats_Truman [wikipedia.org] Daily Tribune announce that Dewey won. Remember President Dewey? So what happened? First ever telephone poll. In 1948, who owned telephones, a rather new fangled technology? That's right, wealthier people. And who do stupid rich people vote for? Right! Republicans! Even in 1948 (some things never change). Classic example of a prediction based on a biased sample. (This is also why, nowadays, when rightwing newspapers publish poll results, they are not trying to report on the election, they are trying to influence it, just like Karl Rove on the Fox Armaggeddon Channel, when he had the "real maths".)
So the only point being made in the Fine Article is 1) landlines are no longer a reliable way to sample public opinion, and 2) polling may no longer be a reliable way to sample public opinion. Now knowing both of these facts, I am sure that professional pollsters will have no problem producing product for us in the forseeable future. It will just be a question of whether it is pull polling, or push polling, or whether advertising is no longer a viable business model for the internet. It's like the guy in the dingy in the hold of the Exxon Valdez at the end of the not-so-great movie "Waterworld", when the flare is dropping down, and he says, "Oh, Thank god."
(Score: 0) by Anonymous Coward on Tuesday June 23 2015, @03:38PM
s/stupid/ignorant/
I'm not sure whether you're genuinely stupid or just ignorant on the difference between those words, though.
(Score: 0) by Anonymous Coward on Tuesday June 23 2015, @03:53PM
Not hating, just pointing out an idiot when one raises its ugly head. If the so egregoiusly down-modded poster had the slighted understanding of sampling, statistics, inductive reasoning, and just plain SCIENCE, they would not have said such a stupid thing.
So it's scientific to simply assume you're right that people's answers are correct. This has nothing to do with denying statistics, sampling, inductive reasoning, or science; it's simply denying bad science built on too may assumptions.
(Score: 2) by aristarchus on Tuesday June 23 2015, @05:15PM
So it's scientific to simply assume you're right that people's answers are correct.
My Gawd, the stupid just keeps on coming! Who said any such thing? I never did, honestly, would I lie to you?
This has nothing to do with denying statistics, sampling, inductive reasoning, or science; it's simply denying bad science built on too may assumptions.
OK, I see there are some cognitive liabilities involved here. The idiot frist poster said any sampling was wrong,
Asking a few thousand people a few questions
Or were we to take the veracity the be the main issue? If we asked more people more questions, would it counteract the mendacity of the source? Or do we have to reduce the dishonesty and then the numbers won't matter? It's all terribly confusing, involving sampling and statistics and correct inductive reasoning, and margins of error. And guess what! Do you not think that pollsters take honesty into account? It is not that hard to confirm your assumptions in election polling, you can just compare your prediction to the official results! If you are wrong, there is something wrong with your methods and/or sources. This is what the Fine Article is saying, they know this already.
To just say, "Grumble grumble Small Sample size! Grumble grumble, People could lie! So Grumble grumble, Polls suck!" sounds just like, exactly like, and nearly indistinguishable from, a climage-change denier. Begin Handwaving, . . . NOW!
(Score: 0) by Anonymous Coward on Tuesday June 23 2015, @05:42PM
The idiot frist poster said any sampling was wrong,
Nope. Notice the "and" in the sentence and you will understand your mistake.
Do you not think that pollsters take honesty into account?
The polls could be manipulating the election results, and the pollsters themselves could be dishonest.
Not only that, but the polls are a useless waste of time.
(Score: 2) by aristarchus on Tuesday June 23 2015, @07:05PM
The idiot frist poster said any sampling was wrong,
Nope. Notice the "and" in the sentence and you will understand your mistake.
Getting rather tiring, but nope on the nope. "And" is a logical conjunction, which means that both claims are asserted to be true. I was only asking for clarification, such that the contrary could be true if one of the conjuncts was not, as in "if the sample size were large enough, would polls be accurate despite respondent mendacity?". Evidently I will not be getting an answer, since
Not only that, but the polls are a useless waste of time.
Why are polls a waste of time? What are they for? Why do some people pay so much money to have them done, and who are these people? Evidently polls are reliable enough for them! So are we going with Karl Rove's "real math", or Nixon's "Silent Majority"? Everyone knows that every agrees with me, so we can ignore polls. Yeah, that's the ticket!
(Score: 0) by Anonymous Coward on Tuesday June 23 2015, @08:17PM
Getting rather tiring, but nope on the nope. "And" is a logical conjunction, which means that both claims are asserted to be true.
It's describing a specific scenario. Learn the way humans actually use English rather than acting like a robot.
Why are polls a waste of time? What are they for? Why do some people pay so much money to have them done, and who are these people? Evidently polls are reliable enough for them! So are we going with Karl Rove's "real math", or Nixon's "Silent Majority"? Everyone knows that every agrees with me, so we can ignore polls. Yeah, that's the ticket!
You're ranting about random nonsense.
Why exactly are polls not a waste of time? Is there some pressing need to know what the election results will likely be before the election that doesn't involve just manipulating voters into voting a certain way? Probably not, or at least not for me. If you want to pay attention to stupid popularity polls, have at it.
(Score: 2) by looorg on Monday June 22 2015, @10:41PM
Why would you assume they all lie or are dishonest? Most people don't go around lying just cause they can. Do you deliberately go around lying to people when asked mundane questions? If they feel they don't want to answer then they don't -- they usually hang up or say no thanks (or you get told to fuck off). Most people don't drag such things out for 10 minutes. But I could be wrong. It's just one of those things or questions that are rarely investigated because it would be so hard to validate or determine if they where truthful or not.
I would assume the disagreement is in regards to you being wrong. Without even getting into the whole statistics probability vs "truth" argument. Asking as few as a thousand people is usually more then enough, decades of public election polling backs this up, I have not checked US numbers lately but for most countries they tend to be in the ballpark -- there are always exceptions but as with most statistics you can usually ignore the extreme outliers. They are very rarely off by big amounts, if they are that tends to mean something is wrong with the method or formula weights or that something beyond the scope of the survey took place. Increasing the amount of people polled usually don't provide much better results, none that can justify the amount of extra time and money spent. Would total surveys (where you ask every one) be better? Sure. But time and money tends to eliminate that as an option.
The biggest problem isn't people having land lines anymore. The biggest problem is the giant stream of polls that are conducted. People just get bored and zone out. To many polls and it's sometime hard to judge if this is a serious poll, some pr-marketing stunt type thing or if it's some organization with an agenda. If you go back to the 70s or before the once that did large surveys was government agencies and universities etc and it was a somewhat clear distinction from the commercial polls (such as if coke is better then pepsi or whatever). These days the overlap is to big and it's hard to tell. The market has been flooded with polling agencies, some that are clearly less serious then others and it's dragging everyone and everything down.
As mentioned in the article telephone polls are loosing efficiency due to cellphones. But also because people find it annoying. They can't seem to tell the difference between serious polls and telemarketers and (scam) pollsters. The growth in the market has lead to an erosion of trust. So instead of having to decide whom to answer you make the easy choice and tell everyone to fuck off. They are also interrupting your daily life by demanding answers over the phone. So some of the older alternatives are coming back again with mailings (it's expensive and the return or answer frequency isn't much better but you can usually ask a large amount of questions -- which is also a drawback; people don't want to answer 20 pages filled with questions about everything under the sun) and personal interviews where you visit the people and ask them in person (problematic if you are asking about personal preferences or sensitive matters -- and it's obvious really expensive).
The biggest problem today, if you ask me, is probably all the small "self-recruited" polls; where people sign up to be polled. They are filled with people on various agendas and the results tend to reflect that. Easy to do and they also tend to give the answers you want. But in the internet and news-cycle they get just as much room as serious polls. People just can't tell the difference, anymore, between good and bad polls.
(Score: 1, Interesting) by Anonymous Coward on Monday June 22 2015, @10:55PM
Why would you assume they all lie or are dishonest?
Not assuming that people gave honest and accurate answers doesn't mean you assume that people are liars, so that's a straw man. Science should be as objective and rigorous as possible. "That would be hard to do." isn't a valid excuse for bad science.
Asking as few as a thousand people is usually more then enough, decades of public election polling backs this up
Or perhaps it's just a means of manipulation, even if unintentional.
I would assume the disagreement is in regards to you being wrong.
There's your issue.
I don't really pay attention to *any* of these silly polls or surveys, much like I don't pay attention to the social 'sciences'; they're meaningless garbage.
(Score: 4, Insightful) by looorg on Monday June 22 2015, @11:25PM
Science should be as objective and rigorous as possible. "That would be hard to do." isn't a valid excuse for bad science.
In most cases science tends to be as objective and rigorous as is possible with the limitations given. There is very little free roaming science around that can do whatever it wants beyond the boundaries of time and money. You want or long for some imaginary world of absolutes that just isn't there and probably hasn't been around for aeons. Most scientific fields today relies heavily upon statistics and modelling and not just the social sciences that you seem to have massive trust issues with for some reason. You have clearly singled it out for your distrust while at the same time ignoring the abundance of statistics used in most other fields. Even if you remove the social sciences from the equation most of the hard or real sciences (whatever they are, we clearly don't share definitions) don't follow or adhere to your strict ideals for objectivity, rigor or absolute answers either.
If you have any suggestion about how to do cheap, instant and accurate polling I'm fairly certain the world or statistics would be eager to hear about them. As with most things time and money does matter.
I don't really pay attention to *any* of these silly polls or surveys, much like I don't pay attention to the social 'sciences'; they're meaningless garbage.
So you don't believe in statistics, social sciences and polling. But yet you had to start a conversation about it. You should probably have put that in your first post and it would have been tagged for trolling instead of disagreement. Shame on me for falling into your trap.
(Score: 0) by Anonymous Coward on Monday June 22 2015, @11:43PM
In most cases science tends to be as objective and rigorous as is possible with the limitations given.
When the accuracy is questionable, the study should say as much. People simply shouldn't assume things to be true just because it would be difficult to investigate much further. The problem is a combination of bad media reporting and, at times, even the ones who conducted the poll are dishonest.
You want or long for some imaginary world of absolutes that just isn't there and probably hasn't been around for aeons.
Straw man.
Most scientific fields today relies heavily upon statistics and modelling and not just the social sciences that you seem to have massive trust issues with for some reason.
Straw man; statistics isn't the main issue.
Even if you remove the social sciences from the equation most of the hard or real sciences (whatever they are, we clearly don't share definitions) don't follow or adhere to your strict ideals for objectivity, rigor or absolute answers either.
They adhere to the ideal far better, however. And no one is talking about absolutes except you.
The main issues with the social 'sciences' is that they usually deal with extremely subjective issues that are almost impossible to verify or investigate. But that doesn't stop the media or many social scientists from claiming that the results of the studies are objective and highly accurate. That doesn't stop researchers from reaching arbitrary conclusions based on data about subjective matters (i.e. porn makes people callous towards women). If the researchers are honest about the limitations of their studies, then that is a good first step. But I won't call current standards good simply because we don't know of a better way to investigate these issues; that's nonsensical.
If you have any suggestion about how to do cheap, instant and accurate polling
That it would be hard to do it better is not a reason to call current methods good, if that is what you're saying. Do you understand that logic?
So you don't believe in statistics, social sciences and polling. But yet you had to start a conversation about it. You should probably have put that in your first post and it would have been tagged for trolling instead of disagreement.
Ah, I see. Not accepting the social 'sciences' because of their obvious problems and limitations [arachnoid.com] is "trolling" somehow. Does anyone even know what a troll is anymore?
Fine, if we're going to play that game, then I say you're trolling because no one can seriously believe that the social 'sciences' are all that credible. Your move.
(Score: 0) by Anonymous Coward on Tuesday June 23 2015, @11:09AM
You are a perfect example of the current culture of replication bullying. Your defamation of science is unacceptable. We will unmask your IP address from behind whatever hackcode you are hiding it with and sue you. Be glad we don't throw you in jail or the asylum for destabilizing society.
https://politicalsciencereplication.wordpress.com/2014/05/25/replication-bullying-who-replicates-the-replicators/ [wordpress.com]
(Score: 0) by Anonymous Coward on Tuesday June 23 2015, @04:46AM
I don't see why anyone wastes their time on it. If they want my opinion, they can pay me in advance, otherwise, just leave me alone. I turned down a some kind of TV survey once and got a "but it's for the Hollywood people" (likely fake) baffled sounding response from the pollster. Who the f*** cares who it's for?
(Score: 2) by TheLink on Tuesday June 23 2015, @06:31AM
Why would you assume they all lie or are dishonest? Most people don't go around lying just cause they can. Do you deliberately go around lying to people when asked mundane questions?
It's not so much due to people being dishonest, but people being easily manipulated and of pollsters being dishonest and not interested in accuracy:
https://www.youtube.com/watch?v=G0ZZJXw4MTA [youtube.com]
https://www.youtube.com/watch?v=If9EWDB_zK4 [youtube.com]
(Score: 5, Insightful) by Geezer on Monday June 22 2015, @08:20PM
The one inside the polling place. Please use responsibly.
(Score: 2, Insightful) by GlennC on Monday June 22 2015, @09:01PM
Please use responsibly.
That's right, citizen. Cast your vote, R or D, your choice.
Sorry folks...the world is bigger and more varied than you want it to be. Deal with it.
(Score: 4, Insightful) by JeffPaetkau on Monday June 22 2015, @11:25PM
Although that is true, the "one inside the polling place" is not entirely disconnected from the so called "horse race" polls. This is particularly true when there are more than 2 options such as here in Canada or in the US when there is a strong independent candidate.
For example, the NDP never had success in either Alberta or Quebec. However, in the last elections (Alberta provincially, Quebec Federally) they did extremely well. Their victories would not have been possible without the polls showing them as being in contention. Without that information voters could only assume that they would be a "wasted vote" as they had always been in the past. Of course that is a self-fulfilling prophesy. The polls showing them doing better than they had historically, snowballed into them doing even better ... and better ... and better ... until they were seen as a legitimate choice.
(Score: 2, Insightful) by Anonymous Coward on Monday June 22 2015, @08:25PM
A poll asking if they even vote anymore or if they even trust their government.
Cause when you view today's headlines, there just can't be an inkling of trust in the public sphere.
Look who you have for president.
He has a nobel peace prize in one hand and a drone kill list in the other.
I'll tell you who will be president next because it's the next greatest fashion.
Wait for it.....................you guessed it...........................A woman.
(Score: -1, Troll) by Anonymous Coward on Tuesday June 23 2015, @06:49AM
.......................A woman.
Oh dear, this is horrible news! She will surely outlaw penises and order them cutted off on sight! Make little boys too wear pink clothes and tell people to brush their teeth...
We're doomed, I tell you DOOOOOMED!
(Score: 1, Interesting) by Anonymous Coward on Tuesday June 23 2015, @09:24AM
the next greatest fashion
We could do worse than have someone in charge who has given life to a human.
Hell, we have--many times.
I saw a thing the other day that said Carly has zero chance of getting the top spot on that ticket--but that you shouldn't bet against her being offered the VP slot.
It's great that this came up today.
Jill Stein, MD [abetterworld.tv] just announced her candidacy [democracynow.org] for the Green Party ticket.
I remember when The Working Class was gathered around the capitol in Madison, Wisconsin in the dead of winter trying to get the Reactionaries not to sell them down the river.
Jill Stein put on her heavy coat and joined them. [worldwidehippies.com]
Obama never showed.
I remember when the people of Philadelphia were protesting unfair FannieMae home repossessions.
Jill Stein showed up, joined the protest, and got arrested. [cloudfront.net]
No politician from the Big 2 ever showed up.
I remember reading about how when she debated Mitt Romney in the Massachusetts gubernatorial race, The Boston Globe described her as "the only adult in the room". [google.com]
To get an idea of her politics, look at who all makes up her Green Shadow Cabinet. [greenshadowcabinet.us]
Hey, she couldn't possibly do a WORSE job that the bunch of bozos who have paraded through the Oval Office over the last 3 decades.
-- gewg_
(Score: 0) by Anonymous Coward on Monday June 22 2015, @08:55PM
Consider that we don't need to be counting each vote. A _random_ selection of a few is all that's needed. The problem is that polling isn't random.
(Score: 2) by Zz9zZ on Monday June 22 2015, @08:56PM
I never saw the point of election polling. The only effect I can imagine is to get better ratings, or more with more tinfoil hat: shift public opinion. What possible benefit can these polls provide for the actual voter?
~Tilting at windmills~
(Score: 3, Interesting) by Phoenix666 on Monday June 22 2015, @09:37PM
Election polling is part of the "circenses" half of "Panem et circenses." The horse race is the central dynamic of the election spectacle, which means you need a system for score-keeping to keep the leaderboard going. The other crucial aspect of the election spectacle is maintaining the illusion of choice and legitimacy. The Trans-Pacific Partnership trade pact being negotiated in total secrecy has been doing a lot of damage to the former. The Citizens United decision has quite undermined the latter. To me, the real test of this presidential election is whether the most partisan and motivated citizens will continue to accept the legitimacy of the process. Reading the activist progressive and conservative blogs, it seems there is some question about that.
Washington DC delenda est.
(Score: 2) by urza9814 on Tuesday June 23 2015, @01:44PM
They *could* be used to make people less afraid of voting for a third party...
(Score: 2) by kaszz on Tuesday June 23 2015, @04:49PM
Or fake the third party to be insignificant small in number of votes and thus a wasted vote to avoid. So keeping the polls out of the election could actually benefit democracy.
(Otoh it seems then like a choice between three ways to waste a vote)
(Score: 2) by urza9814 on Tuesday June 23 2015, @05:27PM
The election itself already has that effect. But in opinion polls there's no real incentive to do the whole "lesser evil" crap.
(Score: 0) by Anonymous Coward on Tuesday June 23 2015, @03:48PM
If your favourite candidate comes out ahead, you can feel like a winner, while if he doesn't, you can complain about the dumb people who don't understand that your favourite candidate is the right one.
(Score: 2) by kaszz on Tuesday June 23 2015, @05:15PM
Considering the systematic logging and accountability for ever so slight deviation from correct behavior and thinking. Why would people voluntarily put themselves at risk?
"Hey I can see here from our HR consulting partner that in a database record done in 1998 you replied that you voted for a candidate which we don't approve of so we thank you for your application will never call you back - NEXT!".
And then it's the factor. Why spend 5 minutes when I could done something much more rewarding and avoid the interruption? It doesn't pay $$ to answer polls and it won't feed back to the community in any significant way either but the owner of the polling company will profit.