Statistical Support for the Sixth Amendment
The criminal defense world is abuzz over this analysis (PDF), which estimated that juries in a recent National Center for State Courts (NCSC) study reached “accurate” verdicts only 87% of the time. In the 290 cases analyzed, the author, Bruce D. Spencer of Northwestern University, estimated that 10% of convictions were wrongful, and that the chance of an innocent person being convicted was 25%.
Spencer makes it clear: the NCSC study “should not be regarded as a representative sample with respect to jury accuracy.” In fact, the NCSC study doesn’t say anything about cases outside the study: “the statistical inferences do not extend outside the cases in the NCSC study.” I know that it’s fun to do so, but the author deprecates the use of his analysis to draw conclusions about juries overall. My comments on Spencer’s estimates apply only to the 290 cases included in the NCSC study.
Spencer isn’t a lawyer, so he’s looking at the system from an outsider’s frame of reference. What he’s trying to establish is the probability that a factually innocent person was convicted, and that a factually guilt person was acquitted:
The interpretation of the latent variable U requires care: does it refer to correct status under the omniscient viewpoint, or under the procedural viewpoint, or something else? Although we want U to refer to the omniscient view of the true state, the data analysis uses statistical patterns of agreement observed in real cases to estimate the parameters in the models.
(my emphasis). (For the difference between factual guilt and legal guilt, see my posts, Factual Guilt vs. Legal Guilt and Right v. Wrong.)Spencer describes a factually innocent person being convicted as “type I error” and a factually guilty person being acquitted as “type II error.” I’m not interested in type II error — this is the system functioning as it was deliberately designed to do; I don’t consider it error at all. Within the framework of the system, when the jury acquits it’s never wrong because the accused is, by definition, legally innocent.
Type II error is of interest to those who have faith that the system works. I don’t care about type II error, except to note that it is less common than type I error — that is, that it is less probable that a factually “guilty” person will be acquitted than that a factually “innocent” person will be convicted. This would appear to be a systemic violation of the “benign … old … general … maxim … that has been long and generally approved” that it is better that 10, or 100, or 1,000 guilty men escape than that one innocent suffer. (a must-read: Alexander Volokh’s n Guilty Men, 146 University of Pennsylvania Law Review 173 (1997)).
The only data Spencer uses from the NCSC study are: juries’ verdicts, judges’ opinions, juries’ opinions on the strength of the evidence, and judges’ opinions on the strength of the evidence. Spencer first estimates jury error by comparing juries’ verdicts with judges’ opinions. “Simple estimates of jury accuracy can be developed form the judge-jury agreement rate,” he says, “the judge’s verdict [opinion] is not taken as the gold standard.” He makes the assumption that judges aren’t always wrong — that sometimes, when the jury acquits and the judge would convict, it’s a type II error rather than a judging error. So some cases in which the jury acquitted but the judge would have convicted are counted as type II errors. Spencer explicitly assumes that the judge is at least as accurate as the jury. I wouldn’t be comfortable making this assumption, but I’m a lawyer who actually deals with judges every day and not a statistician who might, perhaps, hold them to a higher standard than they deserve. Spencer does give alternate estimates for the probability of a correct jury decision if the judge is more or less accurate than the jury.
If you accept Spencer’s statistical analysis and assumptions, it’s notable how much more often judges would commit type I error — convicting the factually innocent — than juries.
Even if you don’t accept Spencer’s analysis and assumptions, it’s notable how much more often judges claim that they would have convicted than juries actually convict. In 47 of 290 cases, the judges opined that the accused was guilty while the juries acquitted. In 15 of 290 cases, the juries convicted while the judges opined that the accused was innocent. So in those cases in which there was disagreement between judge and jury — in other words, in which it would have made a difference whether the judge or jury heard the case — more than 75% of the defendants had cause to be glad they had jury trials. The judges would have convicted more often than the juries whether the evidence (as rated by judges or by juries) was strong, medium, or weak.
Clearly, if juries get it wrong 13% of the time overall the system is not perfect. But we all know the system isn’t perfect, and that in itself doesn’t mean it isn’t working. If, however, 10% of convictions are wrongful, and if a factually innocent person has a 25% chance of being convicted, the system isn’t working. But this would be an indictment of the system, and not of juries, because whatever can be said of an accused’s chances with the jury, it is clear that they are worse with a judge.
Technorati Tags: jury trial, statistics
About The Author
Is it any surprise to anyone that juries frequently acquit regardless of the evidence? Here in baltimore, conviction rates are through the floor for that very reason. For a jury to wrongfully convict is certainly a shame, but I would wonder what happened in that kind of trial.
On the other hand, I would be very curious if someone could come up with a reasonable estimate of whether a factually innocent person is more likely to be wrongfully convicted by a judge or a jury. It would not surprise me much to find that judges do a pretty decent job of protecting the innocent.
Even more interesting would be an analysis on which judges turn out to be the best. Like how much does experience improve a judge’s rulings? Are former prosecutors, defenders, family lawyers, academics, or private practitioners the best judges?
It would greatly surprise me to fine that judges do a better job of protecting the innocent than juries, so you can’t be a defense attorney. If you are, I want to move to your district because where I’m at a bench trial is known as a “slow plea.” In fact, after my last misdemeanor jury acquital, the judge advised the prosecutor to guarantee conviction by charging a city ordinance violation of disorderly conduct (which doesn’t require a jury trial in my state) and would thus be tried to the judge. The judge told the pros that he would have simply convicted my client of the disorderly conduct, after the jury found him not guilty of the assault charge, and maxed him at six months, depriving my client of any “victory” from the jury’s acquital. What the jduge was truly upset at was because I made him work past 4 pm. Obviously not all judges are this way, but most that I know are more focused on efficiency than effectiveness. In my view judges are part of the bureaucracy, and bureaucracy and justice are like oil and water: if they’re not constantly agitated, they naturally separate. Jury trials allow this essential agitation better than anything else.
You are right. I am not a defense attorney. I am in the pupal(yukyuk) [law school application] stage of that metamorphosis.
I find it greatly unsettling to hear your story. I know a good number of the baltimore trial judges personally, and they tend to be a mixed bag. Some are bright and hard working types, and some are entitled pricks who put in 5 hour days. Most are in between.
I had meant, in my suggestion that judges might protect the innocent better, that juries may be more of a shotgun approach, screwing the innocent just as readily as they anoint the guilty. My limited experience has been of judges who at least bothered to pay attention, who usually knew the lawyers and trial tactics, and who could figure things out.
This prompted my next question of which judges actually turn out to be decent judges. If there were some method for determining, prior to giving them their pretty robes, which of them were going to perform justly. Around here all felony trial judges are elected, but they usually run unopposed after having been appointed to fill vacancies. If there is a good method for finding good judges, then the governor could choose to stock the bench with good folks, or at least pick the winners from the losers out of the larger pile of cronies.
You asked a very good question, one that I ask frequently so didn’t mean to argue with or depress you with a sad story. The good part of that story was having my client’s family all hug me simultaneously when the jury acquitted!
You’re right, a lot of judges are “in between” the idealistic and the extreme example I gave. Ironically this is a judge who is lenient in sentencing, but punishes those who take up his time. IHe’s a judge who imposes a “trial tax” which is sickening but an unfortunate reality.
The fun part of that story is that just before the jury acquitted I had to listen to the judge tell me what a bad job I did. So it was fun to walk out with my client a few minutes later. I knew he was as lazy a prosecutor as he was a judge, so his advice meant nothing anyway.
It’s hard to know what makes a good judge, but the jury system is often maligned by the powerful because it affects their interests the most effectively! Without juries, they can install Alitos all over the place and not even bother with tort “reform” or worry about that pesky Constitution.
Keep asking questions and good luck getting into law school. The story I told might be depressing, but I love my job. It’s fun keeping people who think like this judge in check and I would have a hard time doing anything else!