What the NAS Report Actually Says About You

In 2009, Congress handed the National Academies an awkward job: take a hard, independent look at the forensic science being used to convict people across the country. The panel they built wasn't a comfortable one. Scientists, statisticians, judges, working examiners. They spent two years on it, and ever since, defence lawyers have carried the result into court like a weapon. If a lawyer ever rises to challenge the science behind your discipline, the odds are it's open on the table in front of them.

It's more than 350 pages, and it's dense. You're never going to read all of it, and you don't need to. For you, the person who signs the report and then has to defend it on the stand, almost the whole thing comes down to a single sentence the committee wrote. It's the one that gets quoted most.

Look hard at what that sentence does and doesn't say. It doesn't call your discipline worthless or your colleagues frauds. The committee went the other way, describing a field full of “many dedicated and hard-working professionals.” The cut is narrower than that, and sharper. It's about one specific claim: individualisation, the idea that this print, this tool, this hair came from a single source in the world and no other. For most pattern disciplines, that claim has never been put on a scientific footing. The work can be skilful and useful. It's the certainty bolted onto it in the witness box that runs out ahead of the evidence.

The whole report works as an audit of what your conclusions can carry, how much weight they'll bear before they buckle. It examines your field the way a scientist would, coolly and without flinching. The examiners who hold up best under cross-examination tend to be the ones who've already put themselves through that same audit, in private, before anyone else got the chance.

There's a single line running through the whole report, and getting it right is most of the job. A comparison can answer two very different questions, and one is far bigger than the other.

The first is a class question: could this evidence have come from a group of possible sources? This shoeprint was made by a size-10 trainer with this tread. This fibre is a red acrylic of this type. Conclusions like those usually stand on solid ground.

The second is an individual question: did this come from this one source, and nothing else on earth? That's the leap. And for most disciplines, the committee said it flatly, the science that would justify the leap has never been done.

The committee's cautionary tale is microscopic hair comparison. An analyst can look down the microscope and reliably describe a hair's class features. But without nuclear DNA, the report says, that analyst “can only narrow the field of suspects, not suggest any particular individual.” Years later the FBI went back and re-checked old hair cases against DNA. A sizeable share of the hairs that examiners had once called a microscopic “association” had come from someone else entirely. These weren't lab exercises. They were real cases, with people serving time behind some of them.

So settle which question you're answering before you commit. A confident class statement is good science. Sliding from there into an individual one, without flagging that you've done it, is exactly how an examiner loses a jury.

The report gets unusually blunt about one thing: words. Because words are where the overstatement sneaks in. The committee wrote that the terminology of forensic testimony, words like match, consistent with, identical, to the exclusion of all others, “can and does have a profound effect on how the trier of fact … perceives and evaluates scientific evidence.”

Almost nobody sets out to lie on the stand. It's subtler than that. It's a small, habitual climb in confidence, one rung at a time: from the features correspond, up to it's a match, up to a definite identification, up to to the exclusion of all others. Every rung feels reasonable. Stand at the top, though, and you've claimed something no study on earth can back, and handed the cross-examiner the daylight between what you said and what you can defend.

Epistemological humility is a fancy phrase for a simple idea: claim what you know, and not an inch more. On the stand, it works like armour. A witness who states the limits up front is far harder to rattle than one who claims certainty and then has to defend it inch by inch while a cross-examiner chips away.

For decades, the answer to all this was reassuring: don't worry, the courtroom will catch it. Put a weak method in front of a jury, let a defence lawyer cross-examine the expert, and bad science burns off under the heat. The report even quotes the old metaphor for it, cross-examination as the “crucible” that purifies the evidence. Then the committee checked the metaphor against the record of what happens in real courtrooms.

Why did it fail? Because courts mostly admit a method for one reason: an earlier court admitted it. The report's own phrase is “affirming admissibility citing earlier decisions rather than facts established at a hearing.” The defence lawyer usually doesn't have the science to push back. The jury sees a confident expert. So the method rolls on, court after court, for decades, while nobody asks the one question that matters: does it work at all?

Which leaves the job to you. No courtroom is going to protect your science for you. A judge letting your evidence in front of a jury says nothing about whether it's any good. If anyone is going to be straight about what your method can and can't do, it has to be you, long before you take the stand.

Examiners tend to take this part of the report personally, but when it talks about cognitive and contextual bias, it isn't calling anyone a liar. Your brain is built to read the world in context, and that's exactly what makes you fast and good at the bench. But put a faint, ambiguous smudge of a print in front of that same brain, wrap a story around it, and the story starts filling the gaps for you, usually without your feeling a thing.

And the proof behind it is hard to argue with. In 2006, the cognitive scientist Itiel Dror ran the experiment that nailed it down. He took five experienced fingerprint examiners and gave them a pair of prints to compare. What none of them realised was that they had each identified those exact prints themselves, in real casework, some years before. This time the prints came with a story attached: these were the prints the FBI had infamously attributed to the wrong man after the Madrid train bombings. Nothing about the ridges had changed, yet four of the five reversed the call they'd personally made, and three went all the way to a definite exclusion. The only new ingredient was the story, and it changed what the examiners saw.

And the Madrid case that experiment relied on was real. The wrong man was Brandon Mayfield, an Oregon lawyer. After a latent print turned up on a bag tied to the bombings, an FBI examiner called it a match to him. Then a second examiner agreed. Then a third. The report describes how they “affirmed repeatedly this erroneous decision.” The print wasn't Mayfield's at all. No single careless person caused that. The error rolled downhill, each examiner trusting the confidence of the one before instead of the ridges.

So the answer isn't to try harder to stay objective. You can't white-knuckle past a bias you can't even feel. You build the method so the case never reaches you while it can still bend your call. Reach your conclusion from the mark itself, before anyone briefs you on the suspect. Have a second examiner check your work without knowing what you found, so their agreement counts for something. Look at the crime-scene mark first and commit to what it shows; only then let in the suspect's print, so your answer can't drift back toward the name. Write down what you saw, and when. Do that, and the story has nothing left to act on. It never reaches the part of you that decides.

The report goes discipline by discipline, and those summaries are the most useful pages in the whole document for anyone who has to testify. In the committee's own words, they spell out what your method can hold up and what it can't. That's the exact ground a sharp cross-examiner will dig into, so you may as well get there first.

Find your discipline and read its summary closely. Once you know where your method's support runs out, you can speak with confidence right up to that line, and say, just as clearly, what lies on the other side.

The same limit shows up in every row of that table. Ask a class question, and the science is mostly on your side. Ask the evidence to name one person, one tool, one source, and you've stepped past what anyone has proven. The safer wording in each row does one job: it claims what the method has earned, and not a word more.

For all its weight, the report turns into something surprisingly practical when you boil it down to one examiner's daily habits.

Report your work like a scientist. The committee had no patience for reports that give a bare verdict and nothing else, like “the material was identified as marijuana”, no methods, no mention of how sure you are. A report that holds up shows the lot: how you did the work, what you found, what you concluded, and the sources and magnitude of uncertainty underneath it, the confidence levels, the error rates, the places it could be wrong.

Know where you sit. The committee pushed for forensic labs to be independent of law enforcement, and the reason is subtler than worrying that examiners take sides. It's that working elbow-to-elbow with the prosecution creates a constant, low-grade pull toward confirming what they already believe. You don't get a vote on where your lab sits in the org chart. What you do control is your own procedure: reach your conclusion from the evidence before anyone tells you what the investigation needs it to be, and put it in writing before the pressure arrives.

Use certification, accreditation, and proficiency testing — and know what they have to represent. When an attorney bears down and asks how do we know you're any good?, a certificate on the wall is not, by itself, an answer. A credential carries weight only if it stands for a demonstration: that you have been tested on work like the work in this case, under conditions where you could fail, and you performed. That's the answer that holds up.

Document the basis for every conclusion. Show your reasoning, and you can defend it.

The report's whole demand comes down to one word: precision. Know exactly what your work can prove, and be willing to say it clearly, in open court, out loud. It sounds backwards, but the examiner who can calmly lay out the limits of their own method is the one a jury believes, and the one the science can stand behind.