The last data point from my head-to-head election poll was 30 Oct 2004. My weighted-and-bias-adjusted estimate showed the President Bush with a gap over Senator Kerry on that date of 1.9%. The final election results showed a 2.5% difference between the President and the Senator at the national level.
Every few days you hear that the XYZ poll has President Bush leading (or trailing) Senator Kerry by x percentage points, followed a few days later by the WXY poll reporting exactly the opposite. Confused? So am I--but it might help if you tracked the overall polling trend over time, and knew whether certain polls consistently lean in one direction or the other. I do both of those things in this page. I find:
There is job approval data on President Bush from the first few days of his adminstration but there are only a few months of data on the head-to-head Bush-Kerry polls. I begin with the job approval polls so you can see how bias is measured; however, if you are impatient you can skip down to the section on election polls.
Inspired by the redoutable Professor Pollkatz, I started tracking presidential job approval and disapproval ratings. This graph shows the net approval (that is, the spread between job approval and disapproval) based on data available at PollingReport.com. Superimposed on the plot is an average of all the polls.
Many others have already commented on the striking overall pattern of the President's approval ratings. However, I focus on a less obvious question: is there a general pattern to the polls?
Yes. The plot below shows that the Zogby and Fox polls do seem to show a consistent pattern relative to one another: Fox polls tend to report "high" while Zogby polls report "low."
Do other polls demonstrate consistent patterns? Once again, the answer is yes. Below is a series of panels that isolate each polling firm's polls and plots them against the same overall trend line. As you may be able to see, some polls (among them, Fox, ABC, and Gallup) tend to report high while others (among them, Zogby, Harris, and the Economist) tend to report low.
Some readers may find it easier to spot house effects by focusing on the difference between the trend line and each poll. A plot of those differences is called a residual plot and you can see them in the graphic below. Each of the panels focuses on a single polling house. When the polls for a particular house (mostly) lie above the horizontal dotted line, it means those polls were above the trend line in the above graph. When the polls are mostly below the line, it means they would also have been below the trend line.
Technical note: the average line is a locally weighted scatterplot smoother known as lowess. You can think of lowess as a kind of running mean which puts less weight on observations farther in the past and more weight on recent observations. It also puts less weight on one-time outlier values. The lowess line was calculated with a smoothing span set a few weeks wide--this was chosen because some polls sample at about two-week intervals so any span window shorter than that fluctuates as the periodic polls move into and out of the calculation window. I have tried this same technique with other ways of calculating the average line to make sure the results aren't terribly sensitive to the way the average is calculated: for example, this comparison of the lowess average with the two-week median value shows that two very different averaging methods give essentially the same view of the trend. I've also looked at a cross-validated average where each poll is compared to the average if one excluded that poll--this makes sure that any single poll doesn't unduly influence the overall average (e.g., the Fox poll is compared to the average excluding the Fox poll, and the Zogby poll is compared to the average excluding the Zogby poll); in fact, the trend lines in the above panels were calculated in exactly this way. Because of "local weighting," polls that occur often (such as the Gallup and Fox polls) don't (greatly) overinfluence the line compared to polls that occur less frequently. Under the alternatives I've checked, the findings are substantively similar, i.e., the ranking of polls may get slightly shuffled, but the amount of the lean changes only slightly, typically less than 0.5%. I've also examined poll "error," but this page focuses on the direction and amount of lean so I've omitted that from this discussion.
The previous graph shows that two of the "lowest" polls are the Zogby and Harris polls. A quick look at the questions asked by those two polling firms reveals that unlike the other polls PollingReport.com reports their summaries as a combination of "Excellent/Good" and "Fair/Poor" rather than "Approval" or "Disapproval." Evidently, this way of combining ratings tends to understate the approval-disapproval spread relative to the other polls. One possibility is that some respondents who gave the President a rating of "fair" would also "approve" of the job the President is performing. Note that I am not claiming that the Zogby and Harris polls are wrong; I simply emphasize that they do not appear to be exactly comparable to the other polls.
The weekly polls sponsored by The Economist appear late in the race; nonetheless, they consistently show "low" results. I do not know how the sample population compares to the other polls, but either the sample or the way the questions are asked differ considerably from the others. Note also that, unlike the other polls, the Economist poll is an "internet poll." I have insufficient information to determine why its results are so at odds with the other polls, but we should keep a particularly keen eye on it.
Finally, the graph also shows that the AP poll appears to have differ quite a bit from the average during parts of 2002 and 2003. Interestingly, the sponsorship of the AP poll changed in November 2003: www.pollingreport.com notes that "data from 11/03 and earlier [were] co-sponsored by Cook Political Report" and it appears that AP polls after November 2003 are significantly closer to the overall average than those before. I have no idea what "sponsorship" entails, but since the discontinuity is so obvious, and because I am trying to examine current poll biases, I have decided to exclude AP polls taken prior to December 2003. In contrast, although the Zogby, Harris, and Economist polls appear to be structurally dissimilar to the other polls, it does not appear that their pattern of bias has (significantly) changed over time. I'm keeping them in (but see the technical note above for a quick evaluation of how much difference in the overall average it makes to keep the Zogby Poll in the analysis).
In addition to excluding the AP polls prior to December 2003, I also excluded all polls taken before October 2001. There are two reasons for this: first, the percentage of people "undecided" about the President's job performance was quite high prior to September 11, 2001 (see the next section for a discussion of the "undecideds"); second, there is (the previously mentioned) spike in the two weeks or so right after September 11.
The table below shows the overall difference between each poll and the average, after taking into account these exclusions. The plot is a summary of the table and gives a visual representation both of the size and the direction of the biases. The Zogby and Fox polls are clear outliers (on opposite sides).
|
The Zogby and Harris polls asked (arguably) more nuanced questions than the other polls, with one of the consequences being that there are fewer "undecideds" in those polls than in the others. Could this have an effect on the overall job ratings? The graphic below compares the estimated overall bias effect with the average undecided percentage for each polling house. I exclude polls taken before September 11, 2001; those polls exhibited very high undecided percentages. A vertical dotted line at around 6% marks the overall average undecided percentage across all polls. What you may be able to see from this plot is a weak positive association between undecideds and the house effect. The evidence is inconclusive but it raises an interesting line of investigation: could it be that when "undecideds" are forced to choose, they tend to lean more toward disapproval than approval? In any event, the undecideds could not explain the entire difference between the Fox and the Zogby and Harris polls -- there simply aren't enough of them, so even if some portion of the difference could be due to the way that undecideds lean there is still an unexplained portion that must be due either to randomness or systematic biases in the way the polls are conducted.
The previous discussion was about the President's job approval ratings. What about Bush vs. Kerry matchups? Does the bias in job approval extend to the horserace? National polls have been asking about presidential job performance on a regular basis for years, but head-to-head matchups between candidates only make sense when the final candidates are known. Thus, although we have three-and-a-half years of Presidential job approval polls, there are only several months of polls that measure vote preferences for either President Bush or Senator Kerry, and the first couple of months of this year may have been too early for the polls to settle down. In addition, there are slight differences in measured preference for the President and the Senator depending on whether or not Mr. Nader is included as an option.
With these caveats in mind, let's examine whether there are discernible bias patterns in the head-to-head polls.
Using data at Pollingreport.com, the next graph shows the Bush vs. Kerry spread from each national poll taken since January 2004. As above, a trend line shows the pattern over time. A poll that shows a preference for the President is above the dotted line; a poll that shows the Senator in the lead is below. (If you want to see the separate trend lines for President Bush and Senator Kerry, click here.) It may also be helpful to keep a couple of other things in mind:
The Economist, Harris, Gallup, and Zogby polls are "likely" voter polls; the other polls report on "registered" voters but the Time polls switch mid-stream from likely voters to registered voters and back while the Fox polls switched from registered to likely. Different firms use different ways to determine whether a respondent is a likely voter, but in general it appears that "likely" voters support the President a bit more than "registered" voters. In the graph below, you can see that the Fox polls at the left are red ("registered") while the ones at the right are black ("likely"). The amount of bias or lean for the Time poll will likely be quite variable since not only does it switch between registered and likely voter models, but it also has taken relatively few polls. As we get closer to election day, pollsters start to switch from registered to likely voter polls.
As above, we can see how different each poll is from the average. The plot below shows the Bush-Kerry spread for each pollster over the last four months, with the trend line shown in each panel. You should be able to see that, for example, the Economist's polls tend to fall below the trend line while Gallup's polls tend to fall above it.
I have calculated the average amount by which each poll is either above or below the trend line, and plotted these differences against the job approval differences. That's the next graph, with job approval on the horizontal axis and the the head-to-head comparison on the vertical axis. Polls that cluster around the middle of the plot might be thought of as not terribly biased in either direction. Polls that concentrate in the upper right or lower left might be thought to have consistent biases: a poll located in the upper-right reported more job approval and a larger lead in head-to-head preference for the President than the average poll while a poll located in the lower-left reported less job approval for the President and a polling lead for Senator Kerry.
As you may be able to see, most of the Gallup and Time polls appear in the upper-right while the Zogby and Economist polls appear in the lower-left. The plot below summarizes all of these data by showing the "center" of each poll: for example, the NBC polls tended to match the poll average on the President's job approval, but show slightly more preference for President Bush head-to-head against Senator Kerry than the average.
A few other observations:
An inspection of the head-to-head results shows that in the first few months of this year, when the Democratic nominee was not yet known, there was more volatility in the overall Bush-Kerry spread. I will therefore calculate the average amount of each poll's "lean" only for the last four months. The table below shows the lean and the number of polls used. Keep in mind that while there were many Fox and Gallup polls, there were also very few polls from the firms at the bottom of the table. That means that the amounts (and directions) of lean at the bottom of the table should be taken with an even larger grain of salt than those at the top.
|
Despite large and many grains of salt, it's tempting to use the biases just calculated to adjust polls to see how much of a difference they make in overall trends. Here, then, is a plot that shows how the Bush vs. Kerry spread line differs if we adjust the polls for their average lean; it appears that although any single pollster's polls may be consistently biased, the all-poll average seems to track a little more smoothly. One take-home message may be not to pay too much attention to any single poll's results: focus instead on the overall pattern.
Above I highlighted the difference in President Bush's job performance as reported by the Fox and Zogby polls. Below is a set of panels that show President Clinton's job performance as reported by different polling firms. Note that just as the Fox polls now appear to favor President Bush, they then favored President Clinton; just as the Zogby polls now appear to disfavor President Bush, they then disfavored President Clinton. It appears that the Fox and Zogby polls consistently tend to favor or disfavor the incumbent (compared to other polls) no matter what his political affiliation may be.
My conclusions are three:
