Lies, damned lies, and the statistics that expose them

Sometimes, numbers have an uncanny way of exposing lies.  Consider this tale of two groups: police officers and law school bureaucrats.  With regard to the police, one famous study on police-officer behavior revealed that, before the Fourth Amendment was imposed on the states, the police would simply write in their reports what really happened: they stopped people on the street for no reason, searched them, found drugs, and arrested them.  In fact, the police admitted to this in 33 percent of their police reports.  Only in 14 percent of their reports did they write that the drugs magically fell out of the defendants’ pockets.

But after the Fourth Amendment was imposed on the states, the police were no longer permitted to simply stop and search people at will.  So did drug arrests go down?  No.  Arrests remained the same, except that police could no longer write in their reports that they were stopping people and searching them for no reason.  So the percentage of reports where drugs magically fell out of the defendants’ pockets instantly increased to 50 percent (from 14 percent), while the percentage of reports that admitted to actively searching the suspects fell to 5 percent (from 33 percent). 

Obviously, drugs didn’t magically start falling out of suspects’ pockets at the very same time that it became illegal for the police to stop and search people with no justification.  What was happening was that the police started lying.  These lies are now widely accepted as part of the criminal justice system.  Judges routinely rubber-stamp the lies, and the police even admit to telling the lies.  They’ve even coined a phrase for when they lie under oath: “testi-lying.”

Now let’s shift gears to law schools.  Consider the case of law school bureaucrats that report the employment statistics of their school’s graduates.  These numbers are important because if graduates are able to land jobs, the school’s ever-important US News ranking will go up.  (The US News ranking is so important that law schools and colleges have been caught fabricating numbers in order to boost their rank.)  So in order to paint a rosy picture of their graduates’ employment outcomes, some schools have categorized a huge percentage of their unemployed graduates as not seeking employment.  In fact, 35 schools categorized more than twice as many of their unemployed graduates as not seeking employment than seeking employment.

There are two possible explanations for this.  One, the schools like to categorize their unemployed graduates as not seeking employment because that categorization doesn’t hurt the school’s US News ranking.  Or two, it is possible that large numbers of law school graduates truly decided they didn’t want to work at all, despite their staggering debt loads and their desire to eat and pay their rent.  So which answer is correct?  Once again, a before-and-after statistical comparison is helpful.

Let’s take the recent point in time when the US News changed its ranking methodology and announced that it would simply have one category—“unemployed”—regardless of whether the unemployed graduates were seeking employment, or, as alleged by the law schools, were sitting on their duffs, doing nothing, and refusing to look for work.  Once the US News made that change, “the number of schools that reported having more than twice as many unemployed not-seeking graduates as unemployed-seeking fell from 35 to 4.”

Of course, just as drugs didn’t magically start dropping out of defendants’ pockets, unemployed graduates didn’t magically start looking for jobs only when the US News changed its ranking criteria.   Further, I’ll bet that not a single unemployed law school graduate even knew of, or cared about, this change in the US News methodology.  Law school bureaucrats, on the other hand, both knew and cared.  Obviously, what happened was that once the US News changed its ranking criteria, law school bureaucrats had no incentive to continue lying about their unemployed graduates’ desire (or lack of desire) to seek work, as that particular lie no longer produced any benefit for the law school. 

Ahhh . . . statistics.  Sometimes, the numbers reveal the truth.