The Two Questions PCAST Will Ask Your Method

In 2015, President Obama put a narrow question to his own Council of Advisors on Science and Technology. The 2009 National Academies report had already exposed the problem. What more, on the science side, could be done to make sure the forensic evidence going into court is valid? The answer arrived in 2016: a short, brutal document written by some of the country's leading scientists and statisticians. Defence lawyers carry it. Judges cite it. And sooner or later it gets read at you on the stand.

PCAST found two holes. First, nobody had ever written down the scientific standard a forensic method has to clear. Second, nobody had gone method by method and checked. So PCAST did both, across DNA, bitemarks, latent fingerprints, firearms, footwear, and hair.

Your experience is not the standard, and that stings. Your years of casework, your certifications, your accreditation, the consensus of your whole profession: PCAST says flatly that none of it can stand in for evidence that the method itself works.

Strangely, that should come as a relief. PCAST isn't out to call you incompetent. It lifts the burden of proof off your shoulders and drops it onto the method. The question stops being are you good at your job? and turns into has this technique been tested, by real studies, so we know how often it fails? Your CV can't answer that. It was never built to.

The whole report turns on one move: splitting that question in two. Get the split right and almost everything else clicks into place behind it, the verdicts, the error rates, the rules about what you can say. So that's where we start.

This is the single most useful idea in the whole report. PCAST says scientific validity comes in two separate kinds, and a method has to clear both before your conclusion means a thing.

The first is foundational validity, and it asks: is this method valid at all, for anyone, ever? Has the technique itself been shown, by empirical studies, to be repeatable, reproducible, and accurate, with an error rate that's been measured rather than assumed? This is about the method, not about you. If the answer is no, it doesn't matter how careful you were in this case. There's no science under your feet to be careful on.

The second is validity as applied, and it asks: did this examiner apply that valid method validly, here? Has testing shown you can do this task accurately? Did you do it right this time, with notes another person can check? Did you state the error rate straight and stay inside what the studies support?

The order matters, and cross-examiners know it. Foundational validity comes first, because validity as applied is meaningless without it. There's no point asking whether you applied a method carefully if the method has never been shown to work. You'd just be carefully applying nothing. PCAST ties this straight to the law: foundational validity maps onto the Rule 702 demand for “reliable principles and methods,” and validity as applied onto the demand that the expert “reliably applied” them to the facts of the case.

So before your next report leaves the building, run your own work through both gates. Can you name the studies that show your method is valid at all? And can you show, with proficiency results and reviewable notes, that you applied it validly this time? Answer both, straight, in private, and nothing counsel reads at you on the stand will be news to you.

So foundational validity needs evidence. What kind of evidence? PCAST is exact about this, and it pays to know precisely, because vague answers here are exactly where confident-sounding experts fall apart.

Some methods are objective: every step quantified, almost no human judgement, like DNA analysis of a single-source sample. Those you can validate piece by piece, measuring the accuracy of each step. Most pattern disciplines aren't like that. They run on a trained examiner's judgement, which features matter, whether two patterns are “sufficiently similar.” And PCAST's point is that you can't prise that judgement open and inspect it. It happens inside the examiner's head, out of sight.

The logic is simple and merciless. If you can't open the box, you measure what comes out of it. You hand a crowd of examiners a stack of comparison problems where someone independently knows the right answer, and you count how often they get it wrong. That count, the error rate, and the false-positive rate above all, is the science. Without it, PCAST says, an examiner's statement that two samples are alike “is scientifically meaningless: it has no probative value.”

And not just any study counts. A real validation study has to use samples like the ones in real casework, keep examiners blind to the answer, lock its design in before it starts, be run by people with no stake in how it turns out, publish its data, and be reproduced by other groups. That's a high bar. It's meant to be.

Then PCAST ran each discipline through the test and published a verdict. This is the table a cross-examiner keeps open. Knowing exactly where your own method came out, and the precise words PCAST used to say it, is what lets you testify with confidence up to that line and state clearly what lies past it.

And look at the spread, because PCAST didn't condemn the lot. It graded. Single-source DNA passed outright. Latent fingerprints passed too, though with a real false-positive rate that has to be disclosed. Firearms came close and fell short, for want of a second study. Footwear source-identification, bitemarks, and the hair-analysis claims it reviewed didn't clear the bar at all. Find your own row.

One more line from PCAST worth carrying out of here. A method not yet shown to be foundationally valid is, in its words, “presumptively not established to be foundationally valid.” The burden sits with the science to show the method works; it doesn't fall on the critic to show it doesn't. Silence isn't a pass.

Latent fingerprints earn their own section. They're the most instructive case in the whole report, and the one most likely to get misremembered in your favour.

The good news first. PCAST found latent fingerprint analysis to be foundationally valid, largely on the strength of the FBI Laboratory's black-box studies, which the report openly applauded. That's a real pass, hard-won. After decades of the field being “hailed as infallible” with nothing behind the claim, examiners went and did the empirical work, and it held up. You can stand on that.

Then comes the catch that makes fingerprints the perfect teacher. Valid doesn't mean error-free. PCAST measured a false-positive rate that's real, and almost certainly higher than the jury in front of you imagines.

PCAST put numbers on it: a false-positive rate that “could be as high as 1 error in 306 cases” in the FBI/Noblis black-box study (Ulery et al., 2011), and as high as 1 in 18 in the Miami-Dade Police Department study (Pacheco, Cerchiai & Stoiloff, 2014). Be precise about what those figures are: they are PCAST's upper-bound estimates from the two studies it credited, not anyone's measured everyday rate — and if you cite them, cite the studies, not just the report. Whatever you privately believe your own accuracy to be, the validated rate for the method is the one PCAST says belongs in front of the jury. State it directly. Don't soften it.

And validity as applied still has to be earned in your case. PCAST named the open problems for fingerprints in particular: confirmation bias (examiners revising the features they first marked once a matching print is in view), contextual bias (judgement nudged by irrelevant case facts), and proficiency testing that needs to get harder. The safeguard it pressed is specific and cheap. Work and document the latent before you lay eyes on any known print, and note anything that influenced you during the comparison. Conclude first. Let the comparison print in second.

PCAST didn't stop at methods. It went after language, and not gently. It handed judges a list of phrases they should never permit, because each one claims more certainty than any study can back. If those words live in your usual vocabulary, they're now a published target.

The core principle fits in a sentence: once a method is valid, your claims about its accuracy have to track the empirical evidence, and “statements claiming or implying greater certainty than demonstrated by empirical evidence are scientifically invalid.” That sweeps up a lot of comfortable courtroom phrasing. PCAST even reframed the loaded little word match. It prefers “proposed identification,” precisely because match smuggles in a certainty the data don't earn.

And PCAST never asks you to hedge into mush or to refuse an opinion. It wants the opinion your validation studies support: the error rate, the false-positive rate, the limits, and then a dead stop on the line. There's nothing weak in that. Done right, the error rate is the most credible sentence you'll say all day, because it's the only one you can prove.

One day counsel will lift this report, find your discipline, and read PCAST's words back to you in front of the jury. That moment is survivable, but only if you got there first, by reading it yourself.

Lead with the two-part test. Asked whether your method is valid, split the question out loud: foundational validity (has the method been shown to work?) and validity as applied (did you apply it right here?). Answering inside the framework shows the jury you understand the standard instead of fearing it.

Name the error rate before you're made to. If your discipline has a validated false-positive rate, as fingerprints and the single firearms study do, say it directly. PCAST said examiners should report it. Offering it up yourself is the move of a witness with nothing to hide.

Don't get hostile about the report. Read at you, defensiveness sounds like bias; calm familiarity sounds like integrity. You can disagree with a specific point on stated grounds, as long as you're meeting the document as a colleague and not an enemy.

Get the verdict exactly right. If PCAST passed your method, say so, then give the error rate, because the pass came bundled with it. If PCAST found your method short, don't disguise it. Describe what it can still carry, class characteristics, exclusions, and decline the source identification it can't. Either way, you're standing inside the science.

And keep hold of what PCAST wants. Not silence, not paralysis. Better testimony. Methods that have been tested. Claims that fit the data. Examiners who know exactly where their evidence runs out.

Out of the whole report, the two questions are the part to carry. Is the method valid at all? and Did I apply it validly here? PCAST built everything else on that single split, the verdicts, the error rates, the forbidden phrases, all of it. Walk into the box holding those two questions, and you become exactly the witness PCAST was written to make possible.