Dem
51
GOP
49
Yesterday, we had an item about how Donald Trump's legal troubles appear to be affecting his political prospects. (The short version: He's still in OK shape, though there are signs his support might be fraying at the edges.) Today, we want to talk about a different, albeit related, type of poll you should expect to see a lot of in the next year, namely people weighing in on Trump's guilt/lack thereof, what his punishment should be, etc.
Our case study here is going to be the latest (pdf) from Harvard CAPS-Harris poll, which produced the eye-opening result that, to borrow RollingStone's headline: "Majority of American Voters Shrug at Trump Indictment. 53 Percent Want Him Pardoned If Convicted." We took a closer look at the poll and (spoiler alert!) we were not impressed.
There are at least a couple of very big problems facing any pollster who wants to figure out how people feel about Trump's various legal quagmires. The first of these is, of course, ignorance. Even well-informed politics-watchers (e.g., readers of this site) know only part of the case against the former president (and probably only a small part). Most poll respondents are not well-informed, particularly so early in the process. So, their answers are going to be based on their reflexive partisan response, or perhaps their general sense that Trump is a crook/a martyr, or a naive (in our view) belief that one grand gesture from Biden and the divisiveness of American politics will end. As more information comes out, the general public's responses may become somewhat more meaningful, but in the end, people outside the courtroom invariably have a vastly different understanding of things than people inside the courtroom. And the latter is the only one that matters.
The second problem is that people who talk to pollsters tend to be driven by feelings, particularly when those feelings have no consequences. The most common expression of this is all the people who claim they are going to vote third-party, right up until the time comes to cast ballots, and they decide that would be a wasted vote. It's easy enough to shower Trump with pardons, or demand that he get the death penalty, when the person expressing the viewpoint is not at risk of being responsible for the consequences. It's much harder when a person's opinion has an actual, concrete effect.
In short, when one sees a headline like the one in RollingStone, it's easy to conclude that: (1) they'll never find a jury willing to convict and/or (2) if Trump loses and goes to prison, the majority of the country is going to be upset about that. However, because of the issues we outline above, those conclusions are not actually supported by the data.
Moving on to this specific poll, we've noted several times that Harvard CAPS keeps producing right-leaning results, to the point that there appears to be something very different about either their respondents or, more likely, their model of the electorate. There are numerous clues in this particular poll that we should look askance at their sample/model. To start, while the full cross-tabs have not been provided, the breakdown makes clear that questions like "should Trump be pardoned?" were asked of everyone who answered the phone, including people who... hadn't even heard Trump was indicted. Recall what we wrote above about not-so-well-informed respondents.
In addition, there were other results of the poll that suggest something screwy going on with the sample/model. Harvard CAPS says that if the 2024 election was held today, 45% of respondents say they would vote for Trump, 39% for Joe Biden, and 15% don't know or would vote third party. Just about all polls of Trump vs. Biden have the vote pretty close. But not many have it Trump +6. In fact, the only other pollster that has produced a poll in the last month that had Trump doing that well? Rasmussen.
Let's take a look at one more question that raises significant questions about the sample/model. Harvard CAPS asked: "Do you think Joe Biden took a $5 million bribe when he was Vice President, or is that a false charge?" The result is that a staggering 57% said the President took the bribe, whereas just 43% said it was a false charge. We simply do not believe that 57% of the electorate believes Biden took a $5 million bribe. In fact, we do not believe that 57% of the electorate even knows about this claim. It's almost exclusively a right-wing talking point. So, what happened here? Well, as noted, we suspect a wonky sample/model. Another issue is that there was no room for "maybe" or "I don't know;" it was either "yes" or "no." There's also a third problem we will get to in a moment.
Thus far, then, we've laid out why we're leery of polling on Trump legal questions in general, and why we are leery of any results produced by this poll, in particular. Now let's dig a little deeper and take a look at the poll design, which was... deeply problematic. The question about the $5 million bribe was preceded by several other questions about Biden corruption allegations, including one about his son and one about whether the FBI is corrupt. Among those preceding questions (this one came three before the bribe question) was this:
Do you think that the FBI report from an informant alleging that Joe Biden took a $5 million bribe while he was Vice President should be made public or kept secret by the FBI?
That is a very leading way of asking that question, as it could be understood to say that the FBI itself is alleging a bribe was proffered. Keeping in mind that many respondents apparently don't follow the news closely (remember, some percentage weren't even aware Trump has been indicted again), and keeping in mind that people are going to be hearing this verbally and aren't necessarily going to pick up on qualifiers like "alleging," this kind of question pushes the Harvard CAPS poll dangerously into the realm of, well, push polling. Although in the context of survey design, the term that is generally used is "priming."
And how about the question about whether or not Trump should be pardoned? Here is the exact wording:
If Donald Trump were convicted and sentenced to jail for his handling of classified documents, would you support or oppose a president pardoning him in the interest of national unity?
Again, that is dangerously leading, as it's not clear if "in the interest of national unity" is meant as a descriptor of the president's motivations or the respondent's. In other words, the question could very well be interpreted to mean this:
If Donald Trump were convicted and sentenced to jail for his handling of classified documents, would you, in the interest of national unity, support or oppose a president pardoning him?
We don't think that was the intended meaning, exactly, although when you get sloppy about your dependent clauses, and when people are hearing the questions verbally, you're opening up a lot of potential for misinterpretation.
Another problem with this question, which is the linchpin of the "newsworthy" results is, once again, priming. It follows a series of questions about alleged similar behavior by Joe Biden and Hillary Clinton. You can click on the pdf link above to review for yourself, but the most problematic of the questions (seven prior to the "Should Trump be pardoned?" question) is this one:
Do you think indicting a former president for taking classified documents after leaving office, something done by Hillary Clinton, Joe Biden, and other senior officials, is a fair application justice or selective prosecution?
Again, note the poor wording. The question is presumably written to assert that Clinton and Biden also took documents after leaving office. That's not a great representation of things, as it omits a lot of important context. And beyond that, the question could also be understood to declare that Clinton, Biden and others are guilty not of taking documents, but of "indicting a former president for taking classified documents after leaving office." It's nonsense that Clinton, in particular ever did such a thing, but when respondents are poorly informed, and when they hate Clinton already, it's not too hard to effectively hear that question as something along the lines of: "When Hillary Clinton, Joe Biden, and all these other government officials indict former presidents, are they being fair or just persecuting their enemies?"
We don't know exactly what happened here. Did the folks at Harvard CAPS just have a bad day? Maybe, although there's really no excuse for so many badly worded questions that are open to multiple interpretations, not to mention so many questions that prime respondents to give a particular response. It could be a case of bothsidesism, and that all the questions about Clinton/Biden corruption are so the pollster can claim to be "fair." More probable, in our view, is that "surprising" results get more attention, and the poll was crafted to generate such results. That's actually a significant problem in academia, and has led to many notable ethical breaches. Yet another possibility is that the folks running Harvard CAPS are putting their thumbs on the scale for Republicans, the way Rasmussen does. One tends to assume that academics are fair-minded and/or liberal-leaning, but that's not always the case. And we can tell you that Harvard faculty and political scientists are both more likely to be right-leaning than the professoriate as a whole.
The conclusion here is contained in the headline: You really shouldn't trust these polls, regardless of who is conducting them and regardless of what result they produce. Gauging how the indictments are helping/hindering Trump's political prospects is legit, since we can compare his "before the indictment" numbers to his "after the indictment" numbers. But gauging the public's assessment of the legal case against him, and what the ultimate disposition of the case should be? That's just polling porn. (Z)