If 54 percent of people support Hillary Clinton, the survey estimate might be as high as 57 percent or as low as 51 percent, but it is unlikely to be 49 percent. This truism of modern polling, heralded as one of the great success stories of statistics, is included in textbooks and taught in college classes, including our own.
But the real-world margin of error of election polls is not three percentage points. It is about twice as big.
In a new paper with Andrew Gelman and Houshmand Shirani-Mehr, we examined 4,221 late-campaign polls — every public poll we could find — for 608 state-level presidential, Senate and governor’s races between 1998 and 2014. Comparing those polls’ results with actual electoral results, we find the historical margin of error is plus or minus six to seven percentage points. (Yes, that’s an error range of 12 to 14 points, not the typically reported 6 or 7.)