In The Nation, Meehan Crist and Tim Requarth report on a solved-by-forensic-science case that opens a lot of valid questions about the “science” part of forensic science:

Today, Genrich is 55 years old and has been in prison for nearly 25 years for crimes he says he didn’t commit. His latest appeal has been taken up by the Innocence Project, in the hopes of not only freeing Genrich, but getting the courts to recognize recent scientific challenges to forensic pattern-matching techniques that affect hundreds of thousands of people at all levels of the criminal-justice system. In our investigation, we comprehensively reviewed the literature on handheld toolmarks published in forensic trade journals, dug through past legal rulings, pored over nearly 7,000 pages of trial transcripts, and conducted dozens of interviews with prosecutors, defense attorneys, forensic practitioners, judges, academics, and scientists, from Grand Junction to the Department of Justice. What we found was a startling lack of scientific support for forensic pattern-matching techniques such as toolmark analysis; a legal system that has failed to separate nonsense from science even in capital cases; and a consensus among prosecutors all the way up to the attorney general’s office that scientifically dubious forensic techniques should be not only protected, but expanded. With Donald Trump in the White House and Jeff Sessions at the helm of the DOJ, the nominal momentum for forensic-science reform spurred by the two major reports is slowing. Genrich’s case reveals a system that makes it nearly impossible to throw unproven forensic science out of courts and may be keeping thousands of innocent people behind bars.

[…]

Firearm and toolmark analysis emerged out of a national push in the early 20th century to professionalize police investigative techniques at a moment when Americans were particularly enamored with science. Law enforcement borrowed terms from science, establishing crime “laboratories” staffed by forensic “scientists” who announced “theories” cloaked in their own specialized jargon. But forensic “science” focused on inventing clever ways to solve cases and win convictions; it was never about forming theories and testing them according to basic scientific standards. By adopting the trappings of science, the forensic disciplines co-opted its authority while abandoning its methods.

Amid the swirl of new forensic techniques, the courts realized there had to be a gatekeeping mechanism to filter out quackery. In 1923, the DC Court of Appeals provided that mechanism in Frye v. United States. The judges rejected a doctor’s dubious claim that he could use a polygraph to detect when a person was lying from a rise in their blood pressure. In the ruling, the court said that in order for scientific evidence or expert testimony to be admitted, it must be offered by an experienced practitioner making inferences from a “well-recognized scientific principle” that has “general acceptance in the particular field in which it belongs.” In Frye, the judges deemed the scientists in the “particular field” relevant to polygraph use to include psychologists and physiologists—not just polygraph practitioners who would, presumably, be biased toward preserving the technique’s reputation. The effectiveness of Frye in keeping dubious science out of the courts depends on whom judges include in their definition of the “relevant scientific community.” But as the decades wore on, and the forensic disciplines gained influence, judges tended to restrict their definition of the “relevant scientific community” to the forensic examiners themselves. Judges began taking advice on what counted as good forensics from the very people who invented the techniques and made a living off of them.

In the American criminal-justice system, where prosecutors regularly battle defense attorneys over what constitutes valid evidence, judges’ rulings on admissibility are the final word. Once a technique has made it into court and survived appeals, subsequent judges, most of whom have no scientific training and little ability to assess the scientific validity of a technique, will continue to allow it by citing precedent. Forensic examiners, in turn, cite precedent in order to claim that their techniques are reliable science. Prosecutors point to guilty verdicts as evidence that the science brought to court was sound. In this circular way, legal rulings — which never really vetted the science to begin with — substitute for scientific proof. This is Frye’s fatal flaw: Nowhere in this process is anyone required to provide empirical evidence that the techniques work as advertised. Frye aimed to keep pseudoscience out of the courts, but instead has helped create the perfect conditions to keep it in.

[…]

No human endeavor is perfect, yet many forensic examiners claim “zero” or near-zero error rates. In a widely cited 1984 paper in the Journal of Forensic Sciences, bite-mark examiners claimed a coincidental match would occur less than one in 10 quadrillion times. But when actually tested, even the most experienced examiners were wrong about one in six times, and in one study they struggled to distinguish a child’s bite mark from an adult’s. In 2009, the chief of the FBI Firearms-Toolmarks Unit wrote that a qualified examiner will “rarely if ever commit a false positive error (misidentification).” In practice, error rates for matching bullets to firearms can be dramatically higher: In 2008, the Detroit Police Department’s crime lab was shuttered when auditors found that its examiners made one error in every 10 cases. The head of the FBI’s fingerprint laboratory testified that its error rate was one in 11 million—because he knew of only one error in the FBI’s 11 million comparisons—but subsequent tests of fingerprint examiners show error rates ranging from one in 680 to one in 24.