Selection Bias? PolitiFact Rates Republican Statements as False at 3 Times the Rate of Democrats ANS

Here is an interesting article which is trying to imply that Politifact's facts are biased.  They try this three times in this article, but nonetheless, they report what Politifact says: that Regressives lie significantly more than Progressives lie.  If this subject interests you, I suggest reading a few of the comments too.  I have included the first few. 
Unfortunately, the tables don't seem to be copying with their format intact, so go to the site to see the grids. 
Find it here:  http://blog.lib.umn.edu/cspg/smartpolitics/2011/02/selection_bias_politifact_rate.php  
--Kim

Selection Bias? PolitiFact Rates Republican Statements as False at 3 Times the Rate of Democrats

PolitiFact assigns "Pants on Fire" or "False" ratings to 39 percent of Republican statements compared to just 12 percent of Democrats since January 2010

PolitiFact, the high profile political fact-checking operation at the St. Petersburg Times, has been criticized by those on the right from time to time for alleged bias in its grading of statements made by political figures and organizations.

The organization (and now its more than a half dozen state offshoots) grades statements made by politicians, pundits, reporters, interest groups, and even the occasional comedian (anyone 'driving the political discourse') on a six point "Truth-O-Meter" scale: True, Mostly True, Half True, Barely True, False, and Pants On Fire for "ridiculously" false claims.

But although PolitiFact provides a blueprint as to how statements are rated, it does not detail how statements are selected.

For while there is no doubt members of both political parties make numerous factual as well as inaccurate statements - and everything in between - there remains a fundamental question of which statements (by which politicians) are targeted for analysis in the first place.

A Smart Politics content analysis of more than 500 PolitiFact stories from January 2010 through January 2011 finds that current and former Republican officeholders have been assigned substantially harsher grades by the news organization than their Democratic counterparts.

In total, 74 of the 98 statements by political figures judged "false" or "pants on fire" over the last 13 months were given to Republicans, or 76 percent, compared to just 22 statements for Democrats (22 percent).

First, it should be acknowledged that the number of public officials subjected to PolitiFact's Truth-O-Meter lens from each party is fairly even during the period under analysis.

Of the 511 statements put through the Truth-O-Meter test from January 1, 2010 through January 31, 2011, PolitiFact devoted 74 percent of its attention to current and former political officeholders and elected officials (379 statements), 17 percent to ideological organizations and individuals not holding political office (85 statements), and 9 percent to other groups and individuals without a partisan or ideological agenda (28 statements). Another 20 statements came from chain e-mails, public opinion polls, bumper stickers, or "bloggers" generally (4 percent).

For those current or former political officeholders, PolitiFact has generally devoted an equal amount of time analyzing Republicans (191 statements, 50.4 percent) as they have Democrats (179 stories, 47.2 percent), with a handful of stories tracking statements by independents (9 stories, 2.4 percent).

Assuming for the purposes of this report that the grades assigned by PolitiFact are fair (though some would challenge this assumption), there has nonetheless been a great discrepancy regarding which political parties' officials and officeholders receive the top ratings and those that are accused of not telling the truth.

Republican statements were graded in the dreaded "false" and "pants on fire" categories 39 percent of the time, compared to just 12 percent for statements made by Democrats.

That means a supermajority of falsehoods documented by PolitiFact over the last year - 76 percent - were attributed to Republicans, with just 22 percent of such statements coming from Democrats.

As a consequence, Democrats have therefore been presented as much more truthful - with over 75 percent of statements receiving the top three grades of True (16 percent), Mostly True (27 percent), or Half True (33 percent).

Less than half of Republican statements graded by PolitiFact were regarded as half truths or better - just 90 out of 191 (47 percent).

Republicans were also assigned a larger percentage of "Barely True" statements than Democrats, bringing the tally of all falsehoods or near falsehoods in the bottom three categories to 52.9 percent of Republican statements to just 24.6 percent of those made by Democrats.

PolitiFact Ratings of Current and Former Political Officials, January 2010 - January 2011
Party

Pants on Fire
False
Barely True
Half True
Mostly True
True

GOP

23
51
27
30
31
29

Dem

4
18
22
59
48
28

Ind

0
2
0
1
1
5
% Pants on Fire
% False
% Barely True
% Half True
% Mostly True
% True

GOP

12.0
26.7
14.1
15.7
16.2
15.2

Dem

2.2
10.1
12.3
33.0
26.8
15.6

Ind

0.0
25.0
0.0
12.5
12.5
50.0

Data compiled by Smart Politics.

During the last 13 months, the Republicans that have led the way with the largest number of Barely True, False, and Pants On Fire grades are Sarah Palin with eight, Michele Bachmann with seven, and John Boehner, Mike Pence, and the National Republican Congressional Committee with four each.

Whereas Boehner received six "True," two "Mostly True," and one "Half True" ratings during this span, Pence and the NRCC received none in these categories, Bachmann only two, and Palin just four.

What is particularly interesting about these findings is that the political party in control of the Presidency, the US Senate, and the US House during almost the entirety of the period under analysis was the Democrats, not the Republicans.

And yet, PolitiFact chose to highlight untrue statements made by those in the party out of power.

But this potential selection bias - if there is one at PolitiFact - seems to be aimed more at Republican officeholders than conservatives per se.

An examination of the more than 80 statements PolitiFact graded over the past 13 months by ideological groups and individuals who have not held elective office, conservatives only received slightly harsher ratings than liberals.

Half of the statements made by conservatives received ratings of Pants on Fire (12.5 percent), False (16.1 percent), or Barely True (21.4 percent), compared to 41 percent for liberals.

PolitiFact Ratings of Non-Officeholder Ideologues, January 2010 - January 2011
Ideology

Pants on Fire
False
Barely True
Half True
Mostly True
True

Conservative

12.5
16.1
21.4
25.0
14.3
10.7

Liberal

6.9
24.1
10.3
24.1
17.2
17.2

Compiles PolitiFact statement ratings of ideological organizations and individual ideologues who have not held political office or worked for political parties (e.g. commentators and talk show hosts). Data compiled by Smart Politics.

These findings beg the central unanswered question, and that is what is the process by which PolitiFact selects the statements that it ultimately grades?

When PolitiFact Editor Bill Adair was on C-SPAN's Washington Journal in August of 2009, he explained how statements are picked:

"We choose to check things we are curious about. If we look at something and we think that an elected official or talk show host is wrong, then we will fact-check it."

If that is the methodology, then why is it that PolitiFact takes Republicans to the woodshed much more frequently than Democrats?

One could theoretically argue that one political party has made a disproportionately higher number of false claims than the other, and that this is subsequently reflected in the distribution of ratings on the PolitiFact site.

However, there is no evidence offered by PolitiFact that this is their calculus in decision-making.

Nor does PolitiFact claim on its site to present a 'fair and balanced' selection of statements, or that the statements rated are representative of the general truthfulness of the nation's political parties or the elected officials involved.

And yet...

In defending PolitiFact's "statements by ruling" summaries - tables that combine all ratings given by PolitiFact to an individual or group - Adair explained:

"We are really creating a tremendous database of independent journalism that's assessing these things, and it's valuable for people to see how often is President Obama right and how often was Senator McCain right. I think of it as like the back of a baseball card. You know - that it's sort of someone's career statistics. You know - it's sort of what's their batting average." (C-SPAN Washington Journal, August 4, 2009)

Adair is also on record for lamenting the media's kneejerk inclination to treat both sides of an issue equally, particularly when one side has the facts wrong.

In an interview with the New York Times in April 2010, Adair said:

"The media in general has shied away from fact checking to a large extent because of fears that we'd be called biased, and also because I think it's hard journalism. It's a lot easier to give the on-the-one-hand, on-the-other-hand kind of journalism and leave it to readers to sort it out. But that isn't good enough these days. The information age has made things so chaotic, I think it's our obligation in the mainstream media to help people sort out what's true and what's not."

The question is not whether PolitiFact will ultimately convert skeptics on the right that they do not have ulterior motives in the selection of what statements are rated, but whether the organization can give a convincing argument that either a) Republicans in fact do lie much more than Democrats, or b) if they do not, that it is immaterial that PolitiFact covers political discourse with a frame that suggests this is the case.

In his August 2009 C-SPAN interview, Adair explained how the Pants on Fire rating was the site's most popular feature, and the rationale for its inclusion on the Truth-O-Meter scale:

"We don't take this stuff too seriously. It's politics, but it's a sport too."

By levying 23 Pants on Fire ratings to Republicans over the past year compared to just 4 to Democrats, it appears the sport of choice is game hunting - and the game is elephants.

Follow Smart Politics on Twitter.

Posted by Eric Ostermeier on February 10, 2011 12:03 AM | Permalink ¤

Comments

Politifact's job is not to search out obscure statements that are not even making waves in the media at large in order to provide party balance. Their job is to fact-check what's being said, and often controversial and questionable statements by Republicans have driven the debate in the past 2 years.

For instance, you are close to suggesting that Politifact should ignore Sarah Palin since she isn't an "officeholder" at all (you suggest that as a criteria for the amount of ratings) -- however, that is patently ridiculous given the influence that even a Twitter message from her gets on the day's news. She and Michelle Bachmann are covered widely for their statements, revered by the right and loathed by the left -- given the press attention it is not only Politifact's option but their DUTY to report on the truthfulness of these statements that are driving media debate. Replacing them with comments from (for example) some relatively anonymous Democratic committee chairman that isn't being covered in the news anyways in order to provide sufficient "rating balance" would just be silly.

In order to prove bias at Politifact, I argue that you would have to demonstrate either 1) systematic and repeated omissions from their ratings of widely heard false statements from Democrats or 2) varied standards for rating statements, according to party. You have failed to do either.

Posted by: Vito | February 10, 2011 8:22 AM

Nonsense.

Republicans have given us "death panels," "Obama is a Muslim," and "Obama is out to destroy America as part of his anticolonialist revenge."

Not to mention "Our intelligence confirms there are weapons of mass destruction in Iraq" and "Saddam Hussein and bin Laden are in cahoots."

Republicans' efforts to preserve the privilege and power of the rich, white, conservative minority know few bounds of truth. Of course Republicans lie more than Democrats, at least on policy issues. Why would we expect that both parties must lie equally??

(Perhaps different party members lie equally when it comes to their personal lives, although given how many prominent Republicans have turned out to have hidden gay affairs/sex scandals/inappropriate relationships with interns, I'm not sure about that.)

Posted by: John S. | February 10, 2011 8:47 AM

Pulitzer = Progressive

Posted by: B Mc B | February 10, 2011 9:00 AM

So your argument is that there must be bias at PolitiFact, based on the assumption that Republicans can't possibly be lying that much more than Democrats. That's the extraordinary claim here, and it's incumbent upon you to prove it, not PolitiFact to disprove it. Unless you can come up with some damning examples of liberals' controversial statements going unchallenged or conservatives' words being twisted, I'm inclined to side with PolitiFact over you.

Posted by: Benjamin | February 10, 2011 11:21 AM