The Burden of Proof Moves Upstream: Validating AI in eDiscovery
Editor’s Note: GenAI has made first-pass review faster, but the practitioners who certify a production still have to show how the model reached its conclusions. This article draws on HaystackID’s September 23, 2026, webcast, “Getting AI Right in eDiscovery: Quality, Validation, and Results,” where moderator Todd Haley and panelists Jeffrey Fleming and Young Yu examined what makes AI-assisted review defensible. The discussion followed defensibility upstream, from the vendor contracts that govern where data travels to the reviewer who clicks “okay” on an AI determination. It also covered the validation work that separates real gains from run-to-run noise, and the new FTC requirements that now reach prompts, precision, and cutoff scores. The common thread is that a team’s record of how it used AI matters as much as the results AI produced.
The Burden of Proof Moves Upstream: Validating AI in eDiscovery
By HaystackID Staff
Picture a merger team receiving a second request with four million documents and a compressed deadline. It runs GenAI across the first-pass population, hits 90% recall against its reviewed sample, and starts to believe the timeline might hold. Then the agency’s letter arrives. It wants the prompt language, precision and recall for each of 35 specifications, and a list of every document where a human reviewer overturned an AI privilege call. The review takes weeks. Proving it might take longer.
That gap, between what AI can do and what a team can defend, is now the central test in AI-assisted review. Capability is no longer the main question. Defensibility is, and teams win or lose it long before anyone draws a validation sample: in the contracts that govern where data travels, in who owns each decision, and in what a reviewer actually does when the model hands over an answer.
HaystackID® leaders took up that challenge in the webcast, “Getting AI Right in eDiscovery: Quality, Validation, and Results,” drawing on their work with clients and regulators. The lesson that runs through their experience: teams that hold up under scrutiny treat their record of AI use as a deliverable in its own right.
Governance Starts With a Data Map
Much of AI exposure, according to expert panelist and HaystackID Managing Director Jeffrey Fleming, comes from architecture and deployment, not model quality. Before a team can trust an output, it needs answers to two questions: where does the data go, and who answers for it when the output is wrong?
The options sit along a spectrum of control. Many public tools retain inputs and use them to improve the model. Self-hosted models give full control, along with full responsibility for updates and drift monitoring. The middle tier, private business instances of familiar models and the legal platforms built on them, carries the most ambiguity, because terms vary by provider and service tier. Procurement and legal have to read each agreement, and IT and security have to confirm the controls hold.
That contract language deserves a skeptical reading. Some agreements bar training but retain data for logs or abuse review, which leaves it discoverable and exposed in a breach.
“A lot of times you’ll see them saying, ‘We don’t train on your data,’ but that is not the same as them saying, ‘No one else can touch your data,’” said Fleming.
Sub-processors warrant the same scrutiny.
The larger risk may already sit inside the firewall. Employees adopt tools on their own, and browser extensions and meeting note-takers spread fast. Vendors switch on AI features in software an organization already licenses, or run AI on the back end and disclose it in a quarterly release note. The liability stays with the organization either way.
“You can’t simply point at the vendor and say it was on them,” Fleming said.
Even sanctioned tools drift; when a private deployment changes its default model, answers can shift quickly, and a team needs to understand why before relying on them.
The answer is discipline, not paperwork. Name one accountable owner instead of a committee where everyone points to everyone else. Inventory sanctioned and shadow AI, document which data may go where, and build the technical means to prove people follow those rules. Plan for failure, whether that means reverting to manual work or adding reviewers. None of it, Fleming said, needs a “90-page binder,” but all of it belongs in the annual audit cycle.
A Click Is a Signature
Many AI tools that legal teams use require a human checkpoint: a person who reviews the AI’s call before it becomes final. That safeguard means little if the person is clicking “okay” on a queue of AI determinations, “widget-factory style,” as Fleming put it, and he expects pending cases to test exactly that point soon.
“If you’re going to have a human in there, they have to actually be doing the review,” Fleming said. “And when they click okay, that human becomes responsible for whatever the AI answer was.”
The profession has already seen the consequences. Attorneys have landed before state bars after filing briefs with fabricated cases and invented quotes because someone decided the draft looked good. In eDiscovery, the exposure lands on whoever certifies the production. Whatever the AI includes or excludes, “somebody ultimately is going to have to sign that, and that person is going to be held accountable for whatever comes from that AI use,” Fleming noted.
In Fleming’s view, that makes explainability the floor, not a feature. If a team cannot describe in plain English how its inputs and model produced a given result, the rest of its program will not survive a challenge.
Validation Begins With Knowing the Margin of Error
Validation rigor, Yu argued, should scale with what the AI is deciding. Pre-flagging privilege markers at collection omits nothing, so the burden stays light. Relevance calls that replace or supplement first-level reviewers are different: opposing parties will want to know how the AI got there. That is more scrutiny than search terms have traditionally faced. Parties typically agree on terms and rarely validate them, which leaves everyone blind to what they missed.
Many AI review tools help by design. They run as purpose-driven closed models for responsiveness, privilege, or PII detection, and they do not learn from a team’s criteria, documents, or outputs. That limitation is a feature: a model that trained on one commercial litigation would carry its lessons into the next, unrelated matter.
The first number a team needs is its own variance. Expert panelist Young Yu, senior vice president, advanced analytics and strategic solutions, HaystackID, has found that running the same documents through the same prompt several times shifts recall or precision by about 1%.
“What that really means is that the model understands the language, but sometimes it’s going to get a handful of documents correct or incorrect,” Yu said.
At that level of noise, a prompt revision that moves recall from 90% to 91% may not be an improvement at all. A team that does not know its variance will chase phantom gains.
From there, the method will look familiar to anyone who has run technology-assisted review (TAR). Humans code a sample, the team sets a baseline prompt against it, then revises and tracks whether recall and precision improve. A control set or elusion test measures what the process left behind. Each step needs human coding. In Yu’s experience, that means roughly 500 to 1,000 documents for the training sample, 385 to 3,000 for a control set, and about 1,500 for an elusion sample. Documents too large for the model’s context window, and pockets of data that resist the workflow, still go to people.
That front-loaded effort catches teams who expected AI to remove the work.
“AI is great. It’s easy to use, but it doesn’t take away any of the lift in terms of fine-tuning, in terms of drafting the language,” Yu said.
The hours once spent on coding protocols and search terms now go into prompts.
Disclosure is where that effort pays off or falls apart. Some requests go far, including demands for prompt language and every human-versus-AI overturn. Prompts, like search terms, arguably qualify as work product, yet parties often share both. Recall targets Yu has seen range from 75% to 90%. The more durable protection is openness about method.
The FTC Is Writing the Protocol
The opening scenario reflects recent FTC practice. As Yu pointed out, the model second request on the agency’s website has not changed, but recently issued civil investigative demands and second requests carry new expectations. For the first time, the FTC defines AI-assisted review, and Yu noted that its scope reaches TAR, continuous active learning, and GenAI. The requirements also appear to favor continuous active learning, according to an October 2026 analysis by Arnold & Porter. Parties must finish collecting from every custodian before training begins, which works against older TAR 1.0 approaches.
The terms are demanding. The above analysis states that the FTC now requires at least 75% recall, within a margin of error of plus or minus 3% at 95% confidence. For the first time, it also sets a 75% precision floor, so three of every four produced documents must be responsive. Yu has seen requests that ask for metrics tracked issue by issue, so a request with 35 specifications could mean 35 sets of numbers. If each issue must clear 75% on both recall and precision, teams could iterate indefinitely, Yu warned.
Parties must now disclose their cutoff score, and it can be no lower than 0.50. The floor can create a separate trap, Yu noted: where a model’s uncertain documents cluster around that midpoint, teams may have to resolve that entire middle band. The agency also requires parties to disclose their prompts, and some requests Yu has seen ask whether AI was used for privilege, even when humans reviewed every privilege call.
The timing rules push the work even further upstream. Parties must make their disclosures at least a week before training or review begins, and they cannot pre-cull with search terms or email threading without the FTC’s agreement. An elusion sample is due at least two weeks before compliance. Where available, parties must also produce a relevance score and rationale for each document.
Many of these terms can be negotiated. Parties can seek modifications from the FTC, and the panelists described similar openings with the DOJ and comment periods on new guidance. Pushing back carries risk, though, as noted by the Arnold & Porter analysis. A party can follow its own reasonable protocol and explain why, but the agency may dispute whether it substantially complied, which can delay a deal’s closing. It also remains unclear whether a court would enforce the FTC’s terms. The strongest negotiating position pairs technical and legal fluency.
“A combination of a data expert and a legal expert has done really well at talking about modifications to these regulations and these modified CIDs,” said moderator Todd Haley, executive vice president, operations, HaystackID. The aim, he said, is to understand what the agency is trying to learn and offer a better way to show it.
Review Doesn’t Disappear. It Moves Up.
GenAI hasn’t eliminated first-pass human review; it’s relocated it, according to Yu. Humans still write the relevance criteria, code the samples, and handle documents the models cannot ingest. Teams that drop first-pass review shift that work to second-level reviewers and subject matter experts, at a ratio Yu estimated between one to three and one to five.
History supports that view. Reviewers did not vanish when review moved off paper.
“Going from that technology to this technology may change what the reviewers are doing or what they’re focused on, but I think it will actually raise the bar of value in that review,” Haley said.
In the end, as Fleming framed it, full automation is a risk-acceptance decision. Someone must trust the model, or the vendor’s, enough to sign the certification. Until confidence reaches that level, a complete phase-out stays out of reach.
That returns the story to the merger team and its letter from the agency. Teams that respond quickly to such a letter won’t necessarily have the best model. They are more likely to have a named owner, contracts they actually read, reviewers who reviewed, a known variance, and a dated record of every prompt revision and the metrics it produced. AI made the review faster. The record makes it defensible.
About HaystackID®
HaystackID® solves complex data challenges related to legal, compliance, regulatory, and cyber requirements. Core offerings include Global Advisory, Cybersecurity, Core Intelligence AI™, and ReviewRight® Global Managed Review, supported by its unified CoreFlex™ service interface and eDiscovery AI® technology. Recognized globally by industry leaders, including Chambers, Gartner, IDC, and Legaltech News, HaystackID helps corporations and legal practices manage data gravity, where information demands action, and workflow gravity, where critical requirements demand coordinated expertise, delivering innovative solutions with a continual focus on security, privacy, and integrity. Learn more at HaystackID.com.
Assisted by GAI and LLM technologies.
SOURCE: HaystackID
Advisory Note: Early in a matter, legal teams need answers before the review protocol is even written. Yet, every early insight still has to hold up once opposing counsel or a regulator starts asking how it was reached. Bridging that gap between early speed and lasting defensibility is where AI earns or loses its place in a case. HaystackID® Core Intelligence AI Case Insight, delivered with eDiscovery AI, brings matter intelligence forward to the moment data arrives: it classifies documents by legal issue at ingestion, prioritizes custodians and data sources in real time, and generates summary memos that cite the documents behind every key fact. Its Case Elements extension maps the key documents, people, and events of a matter, while expert legal oversight keeps a human accountable for the calls that matter. Rather than treating speed and defensibility as competing goals, Case Insight is built to deliver early clarity with a traceable path back to the evidence. Organizations looking to understand their data sooner can connect with HaystackID’s team of experts to learn more.