Understanding the World: The World of Political Polling

Can We Trust Political Polls? Is it useful? Is it reliable? There certainly are problems -- my biased take.

Note: Although I've had experience and done investigation into polling and survey methods, I use ChatGPT for research assistance and ghostwriting. I am a long way from being an expert. Clearly you can study any number of books on this topic. I have worked through such

Author's Preface

A while back, I wrote a little article on my experiences in creating a student handbook for the evaluation of professors at my university. I discussed that I had done some survey work there and in other cases, and I had not formally studied it, but I self-studied by reading textbooks on survey methods.

I present a negative case in this essay, being quite biased against polling methods and results. Opinions will differ.

Recently, I've been seeing a lot of information presented on YouTube regarding polling. Usually, it's quite a biased take by the left or the right. Although I don't like those terms, they're probably useful for now. It's the U.S. election we're talking about, of course. I see people coloring maps and making projections and making great pronouncements about all these things, and I think they're probably all full of it.

This got me thinking about survey methods and polling methods in general. I know they are fraught with problems, so I decided to craft this little essay discussing my observations over the decades about my views of the inadequacy of polling.

As an aside, I occasionally get telephone calls from polling organizations. I used to respond; some of them were from the parties themselves, while others were from pollsters. I've since come to look at all of these with great suspicion and don't bother answering them. I sometimes respond politely, sometimes rudely.

Introduction

Political polling has become a staple of modern electoral campaigns, seemingly providing insight into voter preferences and forecasting outcomes. However, far from being an exact science, polling is, at best, a dubious art, riddled with methodological challenges that often cast doubt on its accuracy and utility. This article critically examines the technical and practical issues surrounding political polling, focusing on its use worldwide, particularly in Canada, while addressing its role in shaping elections, public opinion, and political strategies.

Polling: An Art, Not a Science

Polling is often treated as a scientific process, with rigorous methodologies that imply objective results. In reality, it is far from that. Unlike the hard sciences, where repeatable experiments yield consistent results, polling relies heavily on subjective interpretation, varied methods, and assumptions about human behavior. The art of polling lies in how questions are crafted, who is asked, and how results are interpreted, all of which leave ample room for error, bias, and misrepresentation.

Global Perspective on Political Polling

Polling is not limited to North America. Political polls are conducted across the globe—in South America, Europe, and, increasingly, in parts of Asia. In each region, cultural, political, and technological differences influence the methods and reliability of polling. Canadian examples, such as the fluctuating fortunes of political parties in federal elections, demonstrate how even well-established democracies struggle with the validity and accuracy of polling data. In emerging democracies or countries with less transparent electoral processes, the issues with polling can be even more pronounced, with questionable methodologies or even deliberate manipulation of poll results.

Key Issues in Political Polling: Validity, Accuracy, and Reliability

At the heart of the polling process are three essential concepts: validity, accuracy, and reliability. Each plays a crucial role in determining how much trust we can place in any given poll, and each is fraught with challenges:

• Validity refers to whether a poll measures what it claims to measure. Are the questions designed in such a way that they genuinely capture voters’ preferences, or are they skewed by the phrasing, order, or context of the questions? In political polling, poorly worded questions can lead to responses that don’t reflect voters' true intentions, skewing the results.

• Accuracy refers to how close a poll’s results are to the true distribution of opinion in the population. In political polls, accuracy is often compromised by unrepresentative samples, nonresponse bias, or overrepresentation of certain groups, leading to an inaccurate picture of public sentiment.

• Reliability concerns whether repeated polling yields consistent results. A reliable poll is one where, if the survey were conducted multiple times with different samples, the results would remain roughly the same. However, reliability in political polling is often undermined by shifting public opinion, differences in sampling methods, and external factors like current events that may cause rapid changes in voter preferences.

Precision vs. Accuracy: A Critical Distinction in Polling

Though often used interchangeably, precision and accuracy are distinct concepts that apply differently in polling:

• Precision is about the consistency of results—whether repeated polls, using the same methodology, produce similar outcomes. However, precision without accuracy is meaningless; a poll can consistently report a candidate leading but still be completely wrong if the sampling or methodology is flawed.

• Accuracy is about whether the poll's results reflect the actual opinions of the population. A poll may be precise but inaccurate if it systematically overrepresents certain voter demographics.

Both precision and accuracy in polling are complicated by external factors such as the size of the sample, the representativeness of the sample, and the methodologies used to contact respondents (e.g., landlines, cell phones, or online platforms).

Sampling and Bias: The Achilles’ Heel of Political Polling

The foundation of any poll is the sample—those individuals who are surveyed to represent the broader population. In political polling, sampling bias is one of the most significant sources of error. If a sample is not representative of the electorate, the poll results will be skewed, no matter how rigorously the data is analyzed.

• Nonresponse Bias: Certain groups are less likely to respond to polls, whether due to apathy, lack of access, or distrust in polling. In Canada, younger voters, Indigenous populations, and rural residents are often underrepresented in political polls, leading to inaccuracies in predicting electoral outcomes.

• Overrepresentation: On the flip side, certain groups, such as older voters or more politically engaged individuals, may be overrepresented, which can give a misleading picture of broader public sentiment.

• Question Bias: Even when a representative sample is obtained, the way questions are phrased can introduce bias. Leading questions, vague wording, or the order in which questions are asked can all influence how respondents answer, often without them realizing it.

Political Polling and Its Predictive Power

Political polling is widely used to predict electoral outcomes, but its track record is far from perfect. In Canada, as in the U.S. and other parts of the world, there have been notable polling failures. For instance, in the 2012 Alberta provincial election, polls consistently showed the Wildrose Party leading, only for the Progressive Conservatives to secure a majority government. Similar polling misses occurred during the 2016 U.S. Presidential election, where polls failed to capture late shifts in voter sentiment.

Such failures highlight the limitations of political polling as a predictive tool. Polls offer a snapshot of public opinion at a specific moment in time, but voter behavior is fluid and can change rapidly in response to political events, campaign strategies, or personal decisions made in the voting booth. Moreover, polls often struggle to accurately predict voter turnout, a critical factor in determining electoral outcomes.

Informing, Misinforming, and Disinforming: Polls’ Role in Shaping Elections

Polling does more than predict—it shapes the political landscape by influencing how voters perceive the viability of candidates or issues. This influence can take three forms:

1. Informing: This refers to polls that provide correct and true results. These are polls that accurately reflect the opinions and preferences of the population sampled. In such cases, the poll has successfully employed valid methods, ensured a representative sample, and formulated unbiased questions, resulting in data that can be trusted to give a truthful snapshot of voter sentiment.

2. Misinforming: Misinformation occurs when a poll, despite not being deliberately fraudulent, provides inaccurate or misleading results. This could be due to a variety of factors: nonrepresentative samples, poorly designed questions, or flawed polling techniques. For example, if a poll consistently underrepresents certain demographic groups (e.g., younger voters), its results will fail to reflect the broader electorate, thus misleading the public and political actors. While the pollster may not intend to deceive, the end result is still incorrect information that misguides those who rely on it. This relates directly to issues of accuracy, validity, and reliability in polling.

3. Disinformation: Disinformation refers to polling that is intentionally slanted or biased, often with the goal of manipulating public perception or political outcomes. In this case, pollsters may knowingly employ biased sampling methods, ask leading questions, or even falsify data to push a particular agenda. The poll results are not just incorrect—they are deliberately designed to mislead. Although it’s difficult to prove outright fraud in polling, the possibility that some pollsters might engage in such practices remains a concern, especially in highly charged political environments where influencing public opinion can have significant consequences.

Summary

Given all the issues—validity, reliability, bias, and question framing—how much can we truly trust political polls? While they can provide some insight into voter preferences, their utility is limited by issues of accuracy, precision, sample representation, and human error. In political polling, these issues are compounded by the fact that public opinion is often fluid, and small shifts can significantly alter predictions. Ultimately, the question remains: can we trust polls? The answer depends on how they are conducted and interpreted. For those who believe in the power of data, polls may provide some value. But for those who recognize their limitations, polls should be seen as just one of many tools, and a flawed one at that, in understanding public opinion and predicting political outcomes.

Bibliography

1. The Oxford Handbook of Polling and Survey Methods https://academic.oup.com/edited-volume/34751
   This comprehensive volume explores the interdisciplinary nature of polling and survey methods, covering everything from political behavior to empirical methodologies in survey research. It addresses key issues such as question framing, sampling bias, and the reliability of results.

2. Gideon, L. (Ed.). (2012). Handbook of Survey Methodology for the Social Sciences. New York, NY: Springer. https://link.springer.com/book/10.1007/978-1-4614-3876-2
   This book covers the technical aspects of survey research, including sampling techniques, reducing biases, and dealing with nonresponse issues. It provides a detailed look into methods like computer-assisted surveys and web-based survey designs, which are increasingly relevant today.

3. Rea, L. M., & Parker, R. A. (2005). Designing and Conducting Survey Research: A Comprehensive Guide (3rd ed.). San Francisco, CA: Jossey-Bass. https://link.springer.com/book/10.1007/978-1-4614-3876-2
   This book is a good primer on how surveys and polls are designed, conducted, and interpreted. It explains the statistical and methodological underpinnings of surveys, making it a great resource for understanding the challenges of polling accuracy.

4. Groves, R. M., Fowler Jr., F. J., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). Survey Methodology (2nd ed.). Hoboken, NJ: Wiley. https://books.google.com/books/about/Survey_Methodology.html?id=HXoSpXvo3s4C
   A well-regarded text that focuses on the design, implementation, and analysis of surveys. It covers a wide range of survey methods and is useful for understanding the technical challenges of conducting accurate and reliable political polls.

5. Weisberg, H. F. (2005). The Total Survey Error Approach: A Guide to the New Science of Survey Research. Chicago, IL: University of Chicago Press. https://press.uchicago.edu/ucp/books/book/chicago/T/bo3619292.html
   This book focuses on the concept of "total survey error," examining all potential sources of error in survey research, including question bias, sampling issues, and nonresponse biases.

6. Alreck, P. L., & Settle, R. B. (2004). The Survey Research Handbook (3rd ed.). New York, NY: McGraw-Hill. https://scholar.google.ca/scholar?q=Alreck,+P.+L.,+%26+Settle,+R.+B.+(2004).+The+Survey+Research+Handbook&hl=en&as_sdt=0&as_vis=1&oi=scholart
   This text provides a thorough guide to designing surveys, sampling methods, and analyzing survey data. It includes practical advice for conducting reliable and valid survey research.

7. Dillman, D. A. (2014). Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method (4th ed.). Hoboken, NJ: Wiley. https://psycnet.apa.org/record/2014-34233-000
   This book presents the tailored design method for surveys, focusing on improving response rates and data quality across different survey modes, including internet, phone, and mail surveys.

8. Fowler Jr., F. J. (2014). Survey Research Methods (5th ed.). Thousand Oaks, CA: Sage Publications. https://www.amazon.ca/Polling-Public-Opinion-Canadian-Perspective/dp/0802038190
   This is a practical guide that delves into survey research methods, focusing on how to minimize error, improve sampling, and ensure that questions are valid and reliable for the respondents.

9. Weisberg, H. F., Krosnick, J. A., & Bowen, B. D. (1996). An Introduction to Survey Research, Polling, and Data Analysis (3rd ed.). Thousand Oaks, CA: Sage Publications. https://www.amazon.ca/Polling-Public-Opinion-Canadian-Perspective/dp/0802038190
   This book introduces the fundamental principles of survey research, focusing on polling and data analysis, and is particularly relevant for political polling.
   URL: https://us.sagepub.com/en-us/nam/an-introduction-to-survey-research-polling-and-data-analysis/book9715

10. Turgeon, L. (2011). Polling and Public Opinion: A Canadian Perspective. Vancouver, BC: UBC Press. https://www.amazon.ca/Polling-Public-Opinion-Canadian-Perspective/dp/0802038190
    This book focuses on political polling in Canada, providing a detailed examination of how polls influence public opinion and political outcomes within the Canadian context.