[Webcast Transcript] From Hype to Workflow: Insights from Experts on the Impact of AI on eDiscovery

HaystackID Blog | July 28, 2026

Editor’s Note: Over recent years, most eDiscovery conversations about AI have centered on adoption: who’s using it, how fast, and how much it costs. Less settled is the harder question underneath that: how do you actually defend these workflows once they’re in place, especially as courts start weighing in on where search, culling, and AI-assisted review fit together. The HaystackID® webcast, “From Hype to Workflow: Insights from Experts on the Impact of AI on eDiscovery,” takes a different angle, asking practitioners who’ve been in the room for both eras of the technology to talk candidly about what’s actually working, where courts are landing, and what still needs to be built. The result is less a sales pitch and more a working conversation between people who use these tools every day and don’t always agree on the details. A recent court ruling discussed by the panel offers a useful real-world data point on that defensibility question. Read on for the full conversation and key takeaways.

---

Expert Panelists

+ Esther Birnbaum
EVP of Data Intelligence, HaystackID

+ Philip Favro (Moderator)
Founder of Favro Law PLLC

+ Aleida Gonzalez
Global Advisory Managing Director, HaystackID

+ Jim Sullivan
Chief Executive Officer, eDiscovery AI

+ Cristin Traylor
Senior Director, AI Transformation and Law Firm Strategy, Relativity

---

[Webcast Transcript] From Hype to Workflow: Insights from Experts on the Impact of AI on eDiscovery

By HaystackID Staff

Two years ago, letting AI make final relevance calls on a document review would have raised eyebrows in most legal departments. That skepticism has largely faded, according to panelists on HaystackID’s latest webcast, who described a shift in sentiment around GenAI that has moved faster than what they saw during the early days of technology-assisted review.

Over the course of the discussion, the panel traced how GenAI is reshaping search, review, and early case assessment, and made the case that organizations still running discovery workflows designed before these tools existed are due for a rethink. One thread ran through much of the conversation: the work of document review isn’t vanishing; it’s changing shape.

Birnbaum described this less as review teams being replaced and more as them being reconfigured; large teams of reviewers reading line by line are giving way to smaller groups of subject matter experts who spend their time iterating on prompts and checking quality rather than eyeballing every document.

That shift doesn’t mean the old rules no longer apply. The panel pointed to a recent ruling in Schulte v. LinkedIn Corp., where a court allowed search-term culling ahead of AI-assisted review, as a sign that courts are working to fit these new tools into familiar frameworks. And the validation principles that have long governed review, recall, precision, and elusion testing aren’t going anywhere either; they’re simply being applied to a new generation of technology. Gonzalez returned repeatedly to the importance of governance, pointing to frameworks like ABA Formal Opinion 512 as essential guardrails for using these tools defensibly.

Read the full transcript below and watch the recording to learn about the panel’s take on the Schulte ruling, validation standards, and what AI-driven review means for the future of legal work.

---

Transcript

Mary Mack
Thank you for joining today’s HaystackID webcast, “From Hype to Workflow: Insights from Experts on the Impact of AI on eDiscovery,” hosted by EDRM, the Electronic Discovery Reference Model. I’m Mary Mack, CEO and chief legal technologist for EDRM. Today’s expert panel is led and moderated by Phil Favro, founder of Favro Law PLLC, and includes Esther Birnbaum, executive vice president of Data Intelligence HaystackID; Aleida Gonzalez, global advisory managing director of HaystackID, Jim Sullivan, CEO, eDiscovery AI, and Cristin Traylor, senior director, AI Transformation and Law Firm Strategy at Relativity. We’re recording today’s webcast for future on-demand access. And with all HaystackID webcasts hosted by EDRM, the recording will remain available on the EDRM global webinar channel throughout the next quarter to support your continued learning and reference needs. And before turning it over to Phil for a fuller introduction and the agenda, Holley Robinson of EDRM will share a few brief notes on the webinar console. Over to you, Holley.

Holley Robinson
Thank you, Mary. If you look at the top of your screen, you’ll see the HaystackID logo, which you can click on to learn more about HaystackID. You’ll also see an option to contact Team HaystackID directly, along with speaker bios where you can learn more about today’s presenters. Moving down, you’ll see the Q&A box where you can type in your questions for today’s faculty, and we highly encourage you to do so. We’ll be answering questions during and after the webcast. Below the Q&A, you’ll find today’s resources, including the slide deck, a link to HaystackID’s AI Governance Services, Phil’s article on AI governance, and links to register for the next two HaystackID webcasts; “From Breach to Legal Crisis: The Hours That Define Your Exposure,” on August 18th at 12:00 PM Eastern, and “Getting AI Right in eDiscovery: Quality, Validation, and Results,” on September 23rd at 12:00 PM Eastern. We’d love to have you join us again. Lastly, you’ll see some emojis down at the bottom of your screen. Please feel free to use them and react throughout the webcast. Back to you, Mary.

Mary Mack
Well, thank you, Holley. We are going to get right to the heart of this discussion, and I’m handing it over to your able hands, Phil Favro.

Phil Favro
Thanks, Mary, very much. Thanks, appreciate the kind introduction. I also want to thank Jim, Esther, Aleida, and Cristin for their preparations today. They are well-prepared to cover our topics, insights from experts- truly experts on the impact of AI on eDiscovery. For the benefit of the audience, you can see the agenda. We’re going to cover the state of the technology, where we are, and where we are going. We’re going to cover AI search and review workflows. We’re also going to address what the future holds. I think all of these speakers are excited about the present, but very much interested to see where the technology goes, how it impacts workflows, and how the technology can benefit both parties and their counsel in addressing these issues in litigation, investigations, regulatory issues, and so forth. So let’s go ahead and get started. I do have to cover with you this disclaimer that, of course, this is an educational program. There’s no attorney-client relationship here. None is formed. This is not legal advice. There are no judicial advisory opinions. And as I always say, you’d be foolish to think otherwise. So with that backdrop, let’s go ahead and let’s get started on the state of the technology. Cristin, tell us a little bit about the arc of the technology that you’ve seen over the past five years. I mean, look, you were in big law for 20 years or 20-plus years before you joined Relativity. You’ve seen and handled eDiscovery at so many different levels. Where has the technology come just in the past five years, and tell us how it’s impacting things? And then let’s hear from our other speakers as well.

Cristin Traylor
Yeah. All I can say is, wow, let’s start with that. In the last few years, we went from people starting to experiment with generative AI and play with some chatbots, ChatGPT, to now it is full scale. People, if you’re not using generative AI on your matters, you’re at a competitive disadvantage. We’ve got hundreds of customers using AI for a review, for privilege. Over a hundred million documents have been analyzed. This is not something that just a few people are doing. We have law firms that are standardizing on it, using it, like I said, in every matter, which is really amazing because if we think about where we were before with TAR and active learning, there was adoption, but it was a very slow curve up. And here, we started over here, and we are way up here. Now, I’m not saying that everybody’s doing this, but it is really being used in eDiscovery and litigation and investigations. And like I said, if you’re not, it’s to your disadvantage.

Esther Birnbaum
Yeah, I would add to that and say that I went in the last two years from, “I never want to do a human-first level review. AI is going to completely replace contract review,” to the point where I am now where it is that we have to completely reimagine what our discovery obligations are because of the way that we can access, understand, read data. So I think that it’s a completely different landscape. And it’s not even just about, “Are you using AI?” It’s, “How are we going to practice law in the future?” And the current, but planning for the future.

Jim Sullivan
I’ve seen a huge shift in the mentality of how people are talking about this too. In 2023, there were people that said, “This isn’t going to be a replacement for any review. This is going to augment reviewers.” By 2024, 2025, we stopped hearing that, but then we started hearing that we can’t use it for privilege. And now we’re seeing that, and we’ve now started seeing people who have been running it on privilege reviews as a full-on replacement for logging. So I think the mentality behind it has shifted dramatically. The conversations have changed dramatically. And when you compare it to what we saw with TAR, where the adoption took a very long time, this is much, much different, much, much more aggressive. And I echo everything Cristin’s saying, that the adoption rate has just absolutely spiked faster than anyone could have really imagined and faster than we ever saw with TAR. It’s just a completely different situation.

Aleida Gonzalez
And overall, AI is not a fairly new invention. It’s been around at least since the ’60s. But the key difference now is that we’re seeing it in multiple devices where everyone has access to it, in their homes, at work. With respect to the legal and regulatory requirements, that’s evolving just as much as AI. So now we’re starting to see more requirements. How do we manage this AI? How do we work with it ethically and responsibly? So overall, as the AI continues to evolve in the legal space, we’re going to continue to see laws develop and regulatory requirements as well.

Phil Favro
Appreciate all of your comments. Maybe you can tell us, Esther, just a little bit about how AI has changed eDiscovery processes in the past couple of years. What have you seen with a few more specifics, if you could, Esther? And then from the rest of the group as well.

Esther Birnbaum
Yeah, I would say that the biggest difference that I see is right now the ability to really understand large data sets. And we’re only really starting to scratch the surface with this. We’re able to take a large data set, classify it, understand the topics, understand the categories of data in it. And that’s going to allow us to, even before we say what’s relevant and what isn’t, to see what we’re dealing with. And so I think the question of negotiating search terms or ESI protocols is going to start way earlier. We’re going to start seeing a lot more of an ability to see and understand your data where it’s hosted before we even get down to the downstream discovery. The ability to index data and understand it with GenAI has just completely transformed what we’re able to do. That’s big picture. I mean, obviously what Cristin was talking about earlier in terms of the adoption with privilege review, the adoption with relevance review, I mean, at this point, it’s kind of a no-brainer. And I think as lawyers, our responsibility is to be doing the best work we can for clients. And we’ve proven over and over that the GenAI results are better than human results in a lot of cases. And so if you’re still being resistant to using it, I think that’s detrimental in your practice.

Phil Favro
Yeah. And, interestingly, you mentioned that. I think Esther and Chris, and you’ve both emphasized these particular points; I don’t view this as some sort of rah-rah session. That’s not what we’re talking about, but I sense your enthusiasm for the technology and what it can do for parties and their counsel. And I appreciate you and Cristin both raising that. Maybe Jim, just talk generally. We’re going to get into specifics in a few minutes, but what is AI’s impact on document review workflows and the entire process, even search beforehand? Give us your reactions on this.

Jim Sullivan
I’m going to be bold and just tell you that there is no situation that you should be using contract review teams to review large volumes of documents ever again. There is nothing that a contract review team can do better around data classification than AI. Full stop. I will absolutely accept any challenge on any issue. Document review is the classification of data into essentially buckets based on content, and computers can do that better than people can. And we have already seen the results of many reviews that have exceeded any standards that we’ve ever seen with TAR or the human review team. And that’s not going to get worse. But the advantages are a little bit further than that even. Not only is it better, and that’s a huge part, it’s significantly cheaper and faster by miles. Now, as Esther mentioned, we can understand our data faster. We can learn about our cases earlier. We can decide our case strategies based on information that wouldn’t have been available in the past in the first week rather than waiting for months. So I would just say that early on, there were some initial discussions about how AI might replace that first-level contract review. And a lot of people were skeptical. I have stopped running into skeptics, and most people are seeing that it’s the inevitable future. And now we are just seeing it as a standard part of every single day. We’re seeing reviews that used to take months finished in a day or two with results that are better than ever before at a cost that’s lower, 100% faster, better, cheaper. There’s just absolutely no future for having large groups of people sitting and looking at documents one by one for the sake of data classification.

Esther Birnbaum
I was just going to say that even last week in a very casual conversation, because we’re really cool, Jim and I, and we talk about this. And all the time I said, “I don’t even think the concept of document review is going to be a thing in the future because we don’t need to do a linear review anymore to find relevant data.” When we are able to natural language query a full data collection, it’s going to completely eliminate document review. Period. Full stop. And I know that it’s not that we’re excited about the technology, it’s that we know what the results and outputs and the capabilities are. We’re not rah-rah because it’s our job or we’re excited about it. I mean, every day we say like, “Wow, we can’t believe what we’re able to do with GenAI.” And it has to… I’m going to continue to go back to this. The focus has been a lot about document review, but we really need to reimagine the discovery obligations as a whole because of what we can do with technology now.

Cristin Traylor
And the point that Jim made about case strategy is really important because we can analyze these documents and find the important ones and the hot documents so much earlier, right? Instead of sometimes you would have people reviewing documents for months, you find that smoking gun that could either win or kill your case at the very end. Now you can find that at the beginning, which can help you really inform what we are doing with this matter. Do we need to settle it, or should we take it to trial? Do we have to disclose something? Does someone need to be terminated? You can get all that information at the front end, which I think just makes us sort of reimagine what you can do and, again, how you can practice law.

Jim Sullivan
Right. The thing that I think is really cool too about that is you can get all that information at the front end to understand your liability if you need to fire someone if something’s going on. But most of the discovery costs come from that first level review. And it actually allows us to move the first level review back to see whether or not it’s even going to be necessary. We can identify all the key documents, key themes, seeing everything very early on at a pretty low price point. And then you can decide if you want to even proceed with that review. Because if that case settles and you can avoid the review completely, you can now kind of sit back and say, “We’ll run the review when we have to, but we already know all the facts that we need right now and just what we can meet our production obligations in a matter of a few days at the end if we need to, which has a potential to save just a ton of costs.” But again, it just goes to your strategy can change because now we know so many things and we can get information that was just never possible before.

Phil Favro
This is really terrific. I appreciate the points at the end, Jim and Cristin, that you raised about getting to the key data earlier. That was always the point of ECA. And remember when we started coming out with ECA tools in the late 2000s, the early 2010s, the idea was to get that data quickly, get access to it so you could address things. But maybe talk a little bit in terms of workflows, like how things have changed. How has AI changed early case assessment? And is it really taking ECA to the next level so you can make that assessment? Or again, is this more marketing hype to get any number of new followers and buyers for the technology? Cut through some of the marketing baloney and get into the real workflows and how this is impacting things. Maybe Jim, you could lead. And then Aleida, you can offer some comments as well.

Jim Sullivan
Right. I think I’m seeing more of a bifurcation of the two goals where I need to prepare my case, I have an investigation, I want to learn information, I want to find my key docs, I want to understand what’s going on, and I have to produce all the relevant documents. And before, you had to go through that review to learn about your case, to understand the key documents, to do everything. And now we can do all of those things in an early case assessment context using tools like early case intelligence that will give us all of the information early on that will tell us what we need to put together a case strategy. And then still we have our relevance review tools that are really more designed to just meet that production obligation by determining which documents meet the request for production. But we’re not seeing the first level review used as a point for getting information nearly as much as we used to. And we can get information earlier, we can get information faster, we can get better information than ever before. And we’re seeing in a lot of cases that don’t go to litigation that that’s as far as you have to go. So ultimately, I think that ECA is fundamentally changing the process because we’re able to get more information out of the dataset much faster. And a lot of times we can pretty much get everything we need and then decide if we need to produce; we can go to that relevancy review and determine which documents are relevant, and privilege review for logging and all of that. But as far as information gathering, we are getting to a point, and I think we’ll be there soon, where you’ll be able to gather everything you need for your case in an ECA environment before it even goes into a real review phase.

Esther Birnbaum
Yeah. And if you want to cut through the question of whether it’s marketing or not, we can give you the statistics on how many millions of millions of documents we’ve done this with. It’s not a small number of millions. And then the reality we see is when we have clients who are using our early case assessment tools with GenAI, the response is usually, “We want to use this on every single document we collect in the future because we know that this is a smarter, better, more efficient, cost-effective way to do it.” And we’re seeing clients adopt it on every matter.

Aleida Gonzalez
Technology’s going to continue to evolve, right? And it’s going to make the ECA process far more efficient. But the concern from a governance perspective is, does the organization have everything in place? And if you look at the ABA’s formal opinion 512, lawyers are required to review those terms of use, to understand the terms of use, to review the contracts as well. And as long as you have your processes in place, your governance requirements in place, then you reduce liability, you reduce any risk to the organization. And that’s just a concern: ensuring no matter what technology or how fast it evolves, just ensuring you have your governance requirements in place to at least keep the organization safe.

Phil Favro
Yeah. Thank you, Aleida. And thank you, Esther and Jim. Let’s get into a little bit more detail with these early case assessment workflows. So is AI replacing the existing ECA technologies or is it augmenting the technologies and workflows? Maybe Esther and Cristin, you can talk just a little bit more about how AI works with existing ECA technologies.

Esther Birnbaum
I mean, I think we’re in a completely different era. And honestly, I think that the concept of ECA is actually going to collapse because once you’re able to just query data to get the documents you need in a way that doesn’t require linear review, we’re not going to need a distinction between ECA and review. I know I’m a little bit future-forward, but I really believe in that. And I think that it’s completely changed ECA because like Jim said earlier, nothing is better at classifying data than GenAI. And when you’re able to do that immediately, you don’t need these other ECA tools that we’ve built and have been great. I mean, GenAI just far surpasses the results we’re going to get with any of our older tools. Now there’s always… I’m not saying our older tools are all going by the wayside. There are ways to… We use different tools for hybrid workflows. We’re not saying TAR and CAL are gone. We definitely leverage them all the time, but I think that it’s completely changed the way we work.

Phil Favro
Okay. Cristin, what does the workflow look like?

Cristin Traylor
Yeah. I think we also have to think about the term ECA, which means different things to different people, right? Some people use that term to mean culling, right? So that’s sort of different. How do we cull the data to get to that review population? You can still use deduplication and threading and all of those things because the idea is we’re trying to cull the data set versus more of an early case insights, right? How do I get what Esther and Jim were talking about? How do I get the information about my case that you can use that natural language, question-and-answer, finding out what is in my dataset because I need to make a strategic decision? So I think we need to keep those two maybe a little bit separate in our minds because one is like, how do we get the data down to review, to produce? And the other is, how do I learn about what’s in my data? And sometimes that ECA term kind of conflates the two.

Phil Favro
I agree. I agree. So go into a little bit more detail. We’re going to get into this in just a moment in terms of the culling, but talk a little bit about how the workflows changed now with AI as this ECA technology.

Esther Birnbaum
Our first recommendation for most of our workflows is to take your data set, your collection, depending on what it is, because often search terms are still required by the court. But whatever your data set is, run it through. We have an early case assessment tool that categorizes and classifies your data and gives you a narrative overview of what’s in your data. And no matter what your downstream workflow is, that’s going to create value, whether it’s in prioritizing a review population or identifying the not relevant data at the onset or just having the key fact development already done for you before anything goes to review. So I would say 95-plus percent of the time, our ECA tool, which can be used at the beginning of a case, but really it can be used on any set of data that you need to understand and classify and categorize and see what’s inside of it. So I would say our workflows always start there. Understand your data and then do whatever else you have to with a better understanding that’ll shape the rest of your downstream workflows and strategy.

Phil Favro
Cristin, anything else you want to add to that?

Cristin Traylor
Yeah, I mean I agree with what Esther’s saying. You can use all kinds of our tools to do that, to figure out what’s in your data, who’s talking to who, are there people that maybe… It can actually help you find maybe data that you’re missing, right? Or are there custodians that maybe you need or people that you need to interview and get more information about? So ECA not only can help you narrow down and find issues, but can also help you figure out what pathway do I need to take, where do I need to go, what are these early fact analyses to help me figure out what I need to do next.

Jim Sullivan
And how often in the past did that kind of information come at the end of a case, right, where you’re like, “Oh shoot, I got to collect from this other person who I didn’t know about early on”? And now that’s just an afterthought. We’ve learned about all the people immediately. We have all that information. And so those surprises at the end just used to be normal, right?

Esther Birnbaum
Yeah. And also bringing this back to reality, this is very inexpensive. Your GenAI that you’re using as part of an ECA workflow needs to be inexpensive. And that is what we’re able to deliver. And I think that’s the most important part because this isn’t marketing, this isn’t sales. This is the reality. And at the end of the day, your clients are going to come to you and say, “How can I do this at the lowest price point, but still do it right?” And we can do that now.

Phil Favro
And we’ve got a good question here from the audience, maybe, Aleida, on the governance side. I think this is really good for you. John Blatt asks, “Can this ECA be done SAFELY,” safely in all caps, “through a model like Claude or ChatGPT? Or do you need a tool for it to be done safely?” And he clarifies, “For safe, I’m mostly concerned about exposing documents, prompts, and responses to storage even for 30 days in the LLM provider servers. I’d be thinking more about model weights, having that information integrated into the model weights and becoming a part of the AI model going forward.” Aleida, what’s your reaction to that?

Aleida Gonzalez
Yeah. Overall, it depends. So the tool that you’re going to use should be appropriate for the purpose of your search. So ChatGPT, is that appropriate for ECA? Probably not. If you’re doing a small, minor search, a minor data set, maybe. But again, you’re using this for legal purposes. Did you read the terms of use? Have you actually read your contract for ChatGPT? Is this, for lack of a better term, a closed loop or an open loop ChatGPT? Because if it’s open, obviously anything that you put into it is going to be fed into the model, into the training, and open to the internet in general. But overall, whatever tool you’re using, you must keep in mind, one, you’re required to read the terms of use. You’re required to read the contracts for every tool that you contract for. And as far as using it safely, what does that really mean? Because anything that you put into the… Personally, I don’t trust… I love AI, use it all the time. But personally, I do not trust putting client information into an AI, using client names, any specific or sensitive material or information into an AI. Now, granted, a lot of legal tools probably and most likely have far more safety features than a broad general AI tool such as ChatGPT. So again, from a governance perspective, make sure your tool is appropriate for the purpose of what you’re trying to do.

Jim Sullivan
Well, and I mean, hearing all of that is terrifying, right? You just said a lot of stuff about things that are very complicated about reading terms of service, understanding closed loop, open. Those are things that you can’t just have your organization say, “Yeah, you guys go out and use what you can trust.” And that is why the tools that we have and the tools that are used in this industry are important, because those safeguards are important, and we’ll make sure that we do it. So there are a lot of things that you have to consider: is your data used to train any models? Can the prompts be seen? Are your prompts being stored for 30 days by Microsoft or OpenAI? Are you hitting any guardrails where, if you hit the content filter, that data could potentially go to OpenAI for review? And so all of those things are things that we talk about every day and spend so much time on dealing with security audits. And the things that matter are you have to be sure that your data’s not being used to train any models, you have to make sure that your data is not going to be hitting any content filters that could flag something that would be going to human review where a person could see your documents, you have to be exempt from the retention of your prompts so that those aren’t stored by OpenAI Microsoft. You can’t be using OpenAI or cloud directly. You have to be using them through a tool that’s going to keep the data internal. And then you have to make sure that your endpoints aren’t global. You have to make sure that the LLMs that are using it are going to keep the data in the region that we have. And so all of those things are things that we think about, that we set up, that we make sure that we get all the proper things. I know that Cristin’s team does the exact same thing. These are things that when you start working with a tool, you should be asking these questions, and your security team should be asking these questions to get approval. But that’s why using enterprise-level tools matters, and not just going up and signing up for a free account on ChatGPT and throwing data in. Those types of things are absolutely important, and it’s the things that we all know in the discovery industry. And I’ll just say frankly, a lot of the newcomers in this space don’t understand those types of things, and it has prevented a lot of tech companies from being able to enter the space because they miss those things, whereas companies like Relativity and others that have been in the industry and people that have worked in the industry for a long time understand these types of things.

Esther Birnbaum
Yeah. And I just want to take this one step further. There are security aspects of not training the models, et cetera, everything Jim just mentioned, but it’s also really important to talk about why these tools won’t necessarily be the best tools for you to use because they’re not fit-for-purpose tools. And we build fit-for-purpose tools by understanding our discovery obligations. I’ve been working with Jim for two years to be the lawyer on the technology and say, “What are the guardrails we need to put in? What can and can’t we do? How do we make sure there are guardrails in place so we don’t hallucinate answers, we don’t force our tools to give answers when they can’t?” So it’s not just about the LLM safety and the cybersecurity safety. It’s also about the defensibility of these tools and how they’re built. And Relativity Air builds with that in mind; eDiscovery AI builds with that in mind. So the fit-for-purpose piece of using a GenAI tool is really important.

Aleida Gonzalez
And just a reminder, weaker governance increases reliability.

Phil Favro
Yeah, thank you. Thanks. These are great comments. I want to thank Mr. Blatt for his question on this. It prompted so much good discussion. Let’s segue now to a discussion of search. And we were talking in our pre-webinar, pre-webcast call about, well, search really isn’t a thing with AI-assisted review because with AI, you’re just putting the data through the model. It’s making the culls based on the prompts and so forth, but search continues to be an issue. And Jim, why is that? What are you seeing with search being an issue and other types of methodologies being blended with AI-assisted review?

Jim Sullivan
So when we talk about search, I’ll say basically we’re talking about searching to cull a data set where you’re taking a set of documents, and you’re culling it down through keywords, which is something that we’ve always done and something that we’ve always known as absolutely terrible, but we really just didn’t have a better approach. We’ve seen some cases now that are saying essentially that using search before using an AI relevance tool is appropriate. And that is because it is still unreasonable today to take all of your data and put it through an AI-relevance tool. Every tool in the market is going to be cost-prohibitive to put in that type of volume. And we do need to cull down the data set in some way. And we all know that search has been a way that’s been approved and used in the past. So even though everyone understands that it is very, very bad, it is pretty much the best we’ve had. And that’s one thing that I think is changing. And that’s when we talk about what the future looks like, we’re seeing a lot of things that AI can do better than search, especially in that ECA context, like Cristin mentioned, around when we’re talking about the culling aspect of ECA. That is going to make a big difference because search terms miss foreign language, miss misspellings, it misses context, it misses documents that don’t have text. There are a lot of things that we’ve always known that it is not capable of getting, and that’s where now GenAI can get those things. Misspelling is not a factor. Foreign languages don’t matter. It can understand the context, it can understand slang, it can understand sarcasm. When it uses words that don’t mean what it’s actually saying, those things now can all be captured. And we’re seeing the quality of the results improving dramatically. But as Esther mentioned, it has to come through low-cost ECA tools versus the higher-cost relevancy tools. And that’s taking some time, but I think the future is inevitable on that as well.

Phil Favro
Yeah, Jim, thank you for that. It’s interesting you bring that up because there’s a brand new case out where these very issues were raised. And this is the Schulte case. For the benefit of the group, it’s not in your slides. You are welcome to message us afterward, and we will send you the particular case. And if you want to look it up, again, it’s Schulte versus LinkedIn Corporation. But very interesting in that case. Cristin, I’m going to ask you just briefly about your reactions on this. Judge Laurel Beeler in San Francisco had indicated that the producing party could use search terms to pre-cull the data set. And the requesting party, plaintiffs in that case, had demanded that all data flow through Relativity AI review, your very tool. And of course the producing party, the defendant, did not want to do that because of the very cost issue that Jim raised. And the court said, “Look, looking at cases from the TAR era, I’m not going to require LinkedIn Corporation, the defendant producing party, to do so for proportionality purposes.” And so Cristin, from my perspective, it’s reassuring to see that well-reasoned TAR case law is being relied upon for this purpose, but give us your reaction on this continuing issue of blending search methodologies with AI-assisted review.

Cristin Traylor
Well, yeah, again, it’s what the parties agreed to in this case. So they had agreed they could use search terms. And so there’s really not a lot here to argue about. They couldn’t then just say like, “Oh, now you can’t use search terms.” So I think that’s a little bit different because more along the lines of what Jim’s saying, we need to almost reimagine how we do this culling, this search using generative AI. And so for the parties that maybe they haven’t gotten there yet, and they’re using search terms, then you’re using search terms. You’ve called the data, and now you’re going to use the generative AI tool to review, which is what they did here. And what’s interesting about this case is there wasn’t an argument over using aiR for Review to actually classify the documents and make those responsiveness calls. It was just, how do we get to that population in the first place? And again, you got to look at the ESI protocol and what you agreed to begin with, but the plaintiffs couldn’t find a deficiency in the actual production that they could point back to the search term. So I think that’s kind of where it came out there.

Phil Favro
Yeah. Yeah, I appreciate you sharing that. Jim, appreciate your perspective. So looking at it, we always tested search terms. We always test search terms, hopefully for quality assurance purposes because you don’t know what’s in the data. And so if you’re just putting together these search terms, you’re just spitballing, hoping that throwing darts at a dartboard, you’ll get the bullseye. More likely than not, you’re not going to get it.

Jim Sullivan
I will say that in my career, the number of times I saw people actually testing keywords was embarrassingly low. And oftentimes they were either negotiated blindly, or they had a number of documents they needed to hit, and they found a way to back into that number.

Cristin Traylor
Or if they were testing them in order to find the bad ones, to find the ones that were getting a crazy number of hits, but not to find the ones that weren’t there.

Jim Sullivan
No one’s calculating recall and precision on keywords in 2009.

Esther Birnbaum
I mean, nobody even pretends. Very often the data is not even in a review platform when they’re running search term hits.

Jim Sullivan
We run them before. We run them in processing.

Phil Favro
But are you testing prompts the same way that people should have tested search terms?

Jim Sullivan
Hundred times more, right?

Phil Favro
What’s the workflow?

Esther Birnbaum
There’s so much more interest in it.

Jim Sullivan
Validation.

Esther Birnbaum
I think that AI in these workflows is being held to such a higher standard than search terms ever would. And we are validating every single day. I mean, it’s built into our workflows. We don’t do an aiR for Review project without validating search terms. We don’t do an ECA review without validating… Sorry, I didn’t mean validate search terms, but we don’t do an ECA review without validating the results on the prompt or testing the prompt. We definitely don’t do it on a relevance review. Of course we’re testing. So that never happened before. I mean, search terms, I would say 99.99% of the time in my career, when I saw them, they were completely blind. But I will say that right now, they’re still a necessary evil because we still have to collect in a way that we can use search terms and get to populations that we can manage. So I think we all, in our conversations, agree with this decision in this matter, but I just think the reality is that we have to be real with ourselves. And I think we’re doing a whole lot better now than we were doing with analyzing search terms.

Cristin Traylor
And the irony is what you just said: the technology or the search terms that we knew were less reliable and not getting great things. We weren’t doing all this validation. And yet, as you said, we’re holding this to a huge standard, generative AI to much higher standards when we know we’re getting better and more consistent results.

Phil Favro
And if you remember what Judge Peck had said 11 years ago in the Rio Tinto case, we need to be careful with TAR that we don’t hold it to a higher standard than we would other search methodologies. And yet we are doing that with these different technologies. We certainly did it with TAR, and we certainly are doing it now with AI-assisted review because more questions are being asked. And that leads into another set of questions. So let’s talk about review a little bit, and the review process since that is the heart of AI-assisted review. I think Jim, you had mentioned at the outset that you think large doc reviews with teams of reviewers that these should be done away. Cristin, what are you seeing with your customers at Relativity? Are you still seeing large document review teams, or are these things a thing of the past? And then Jim and Esther and Aleida, I definitely want to hear your views on this as well in how AI is changing this for first pass and second pass review.

Cristin Traylor
Yeah, I agree with Jim. I think that’s where we’re going. And we’re seeing that. So we’re seeing the number of reviewers go down. So you can have aiR for Review do your first pass review, and then you’re going to have a small number of attorneys doing the QC process, doing the validation. You still have people who have to do the prompt iteration and make sure the prompts are right. So the focus now is in different places. It’s not on reviewing the millions of documents with people sitting in a room. And a lot of what the classification tool can do is identify the junk. It can identify that ROT and help us find the SUN so that we can focus on the documents that really matter. And why are we going to spend money having people look at all this, essentially, junk? And so I think Jim is absolutely right. And that’s what we’re seeing as well, is that our law firms, corporations, they’re looking at it and saying, “We don’t need an army of people. We need a small group of very knowledgeable people that can really help us get what we need to do either for production or, again, to get the information to advise the case strategy.”

Phil Favro
What’s the workflow look like? Esther, what’s the workflow look like then? Are we eliminating all first-pass reviewers? Are you using just second-pass reviewers just to sample? I mean, what was LinkedIn doing in that one case we just mentioned? For example, are they doing it consistently with what other companies that you see are using the technology or doing? I mean, bottom line, what’s the workflow look like now?

Esther Birnbaum
Well, I want to be very clear and say that it’s not always easy, even though I 100% believe that this is the way we should be doing it. With a relevant review, if you’re talking binary, is it relevant? Is it not relevant? That’s one prompt that we’re going to need to validate. If we’re talking 15 issue tags, that’s 15 prompts that we will individually calculate recall and precision on and validate and sample and do everything right. And that can take time. And that also takes expertise. It takes people who understand how to prompt. It takes a subject matter expert who understands the subject and the data. And then it requires a review of sample sets of documents. So it’s not like review is gone, but all the work happens there. It happens upfront, and it happens retroactively. So that could take a significant period of time because you are validating a sample set of data. And if you’re doing it against 15 issue tags, that could be very involved. But then once you have your prompts set, the actual review piece- first-level review doesn’t require any more work until you get to the end where you validate the results. So we’re talking about upfront at the top: subject matter experts, GenAI experts, and then you need the people to QC the review. You don’t need a first-level review team. That’s what the GenAI does. That’s for relevance review; the same thing goes for privilege review. I mean, we’re seeing our clients understand that privileged review with GenAI is incredibly valuable because privilege log creation is incredibly time-consuming. And now we don’t need document reviewers to go in and pick options to create a privilege log description. We can get that generated, completely customizable to a matter, and then you only need a QC layer for that. So it’s a completely different workflow. And what Cristin said is it’s a different engagement of who is involved, and it’s not going to ever have to be that massive amount of humans sitting in a room doing linear review.

Aleida Gonzalez
They both touched on an important part of just having the right people to review. So it’s not just about having the proper protocols in place- how to conduct proper search terms, proper prompts. It’s also that human-in-the-loop aspect that would facilitate a good review. So having the right people, hiring the right talent, that’s also an important factor as part of your governance policies as well.

Cristin Traylor
That LinkedIn case that we were just talking about- what they said they were doing again in the case is that they were using it to make final responsiveness calls, and then they were going to do a sample of those calls, and that was going to be their QC. So that is how they were using it. And we have definitely seen a lot of customers using it that way. We’ve also seen some that maybe are using it more to prioritize or triage. I think people are going to use the technology in different ways. It’s just going to depend on what’s right for their matter.

Phil Favro
Yeah. And I’ve heard you guys talk about defensibility. I think Esther, you have mentioned defensibility being a very significant issue. I think all four of you would emphasize defensibility. Aleida, you’ve talked about the governance issues to ensure it’s defensible. And Jim and Cristin, you’ve hit on or touched on this as well. Let’s talk about validation in terms at the end. How has validation changed? We’re not talking about search terms; we’re not talking about TAR. Are you still using the same methods in your workflow? Whether it’s your recall targets or illusion testing? Talk to us a little bit about validation here and what it looks like. Cristin, maybe you can lead out on this, and then we can hear from the rest of the speakers to the extent they have comments.

Cristin Traylor
Yeah, it’s no different from a TAR validation. It’s the same thing. So you can calculate precision, recall, illusion, and get your richness. We have it built into aiR for Review so you can actually run the validation in the platform and get your results, but there’s nothing different about it in that way; you can still get the same metrics that we’ve been relying on for years and years and years. I think the difference is, going back to that workflow, is you start off, you create your prompt, you do some prompt iteration where you’re doing a sample, you’re running the prompt, you’re looking at the results, you’re going back and forth, you’re adjusting your prompt. You can do all that process ’till you feel good about your prompt. Then you can either… And a lot of customers are doing this; they’re pre-validating. So they’re actually running the validation before they run the generative AI tool to see, “Hey, what kind of pushes and recall am I getting? Do I want to go back and iterate more on my prompt?” And then others are doing it at the end. “I’m going to run aiR for Review, classify my documents, and then do an end-stage validation.” But again, the validation itself and all of the principles that go with it are the same.

Jim Sullivan
We’ve been using binary classification with TAR for such a long time. Why change? We’ve built out processes for validation, run them on thousands and thousands of projects, and been surviving all core standards. No reason to do anything differently. We’ve already got a great process that everyone works with. So yes, I agree 100% with Cristin about the exact same process as what we’ve done before. We’re just using different tools to make that classification.

Phil Favro
Well, Esther, you guys have written an article about this. Maybe it’s in our resources that we’ve referred to it. Maybe you can talk a little bit, just briefly, about what you guys have hit on in this article on this particular topic.

Esther Birnbaum
Yeah. So Jim, Cristin, myself, and a couple others collaborated on an article where we go through what we see as the realities in GenAI for eDiscovery. And much of it goes back to what Cristin and Jim said, is that we already have validation, we have processes, we have governance in place that are already built into the law, into our workflows. We don’t have to change that. If you look at it from an objective perspective, we already have a lot. There’s no reason to change our validation when we call them precision are the metrics you need. We go into it in a lot of detail, so I do recommend reading it if you’re interested. But also, it goes back to what we know works and what we already have, and not try to rebuild something that doesn’t need to be rebuilt.

Jim Sullivan
I’ll give you the short-term and the long-term. So in the short term, I think we are going to continue to have better tools that give us access to information faster, better, cheaper. We’re going to continue to evolve where it’s just going to be earlier and earlier. And I think that is absolutely no doubt in my mind that the end of this game is going to be where GenAI does the discovery process, where you have a data set, and you get a request for production and you put that in and it tells you, “These are all of the relevant documents to that request.” And there’s no process of ECA and review as a separate drawn-out process. It’s, “Here are the documents that meet the requirements that you’ve specified from our dataset,” and it’s going to be able to do it much faster, much cheaper. It’s going to be able to provide you any information about your dataset. And basically, getting information out of your dataset is going to be just asking it what you want, and it’s going to give you what you need.

Phil Favro
So is this going to be like a baseball game where now… Remember back in the days with baseball, think about Lou Piniella or Billy Martin who had these wonderful arguments between the managers and the umpires. Now with these automated calls from computers, we’ve lost an element of the game. But maybe we haven’t. Maybe we can just stick to baseball instead of arguing with the umpire. Are we going to be able to eliminate a lot of the contention and actually just get onto the merits of the litigation and get through discovery?

Esther Birnbaum
I think we’ll eliminate the contention in the technical details that don’t matter, the arguing over search terms based on a hit count without doing an analysis, the ESI protocols of allowing GenAI or not allowing GenAI without understanding what you’re actually talking about. We’ll get to a point in the not-so-distant future where data can be indexed in place and you can, as Jim said, natural language query it, and we’ll have ways to validate the answers. And when it goes back to it, what are our discovery obligations? And it’s to produce relevant documents without producing privileged documents. All of the processes we built go back to one simple thing, and we’re going to have a much simpler answer. Do I think that’s going to mean our jobs are gone? No, because technology makes for a whole lot more lawsuits and a whole lot more complications and whole new data types and larger amounts of data. But I think that a lot of the discovery complications that exist now are going to go away.

Cristin Traylor
Yeah, look, generative AI is the great equalizer, right? This is not just good for the producing party. This is really good for the requesting parties. They’re going to get more consistent results; they’re going to get better results. They’re getting higher recall. So they’re getting more of what they need for their case. And they’ve got these tools to help them get through whatever gets produced to them very quickly, raise up those important documents for their case too. So I think once we get down to it, hopefully we’re not fighting over some of the particulars and spending a lot of time and having that TAR tax. Instead, we’re working together collaboratively on both sides of the case because these tools are going to get better results. They’re going to get those relevant documents faster and more consistently.

Esther Birnbaum
And you’re not going to be able to hide behind bad discovery. There’s no more, “I’m nervous about what I might find in the document, so I don’t want to do X, Y, and Z.” There is not, “Let’s use bad search terms to make sure we don’t uncover anything.” It’s going to be more transparent, and it goes back to what we should be doing practicing law, not using data to hide behind what the real purpose is.

Jim Sullivan
I think the second piece of that is if you can access information about your data that much easier, you’re now going to be able to identify issues earlier and potentially before they become litigation. You could have AI, essentially, monitoring for things that create liabilities to your organization before they become litigation, before they become a problem, and preventing some of those risks earlier on.

Aleida Gonzalez
As the tools get better, as they become more effective, we’re going to see greater requirements, whether they’re legal, regulatory, or stricter guidelines. And with that, I think we’re going to see a bit of a shift back and forth between requirements and the involvement of these tools. But regardless, no matter what, you still got to have your protocols in place, your policies in place, and have some redundancy from the validation aspect to data governance, to human oversight, and make sure that all your work and the way you’re managing it is auditable and accountable.

Phil Favro
Good. Good. Thank you, Aleida, for the final word on that. We got a couple of questions from our audience here that we want to follow up on. One, Cristin, I think you were talking about ROT, and we have Erica asking, “I know what ROT is, but did you say SUN or STUN?

Cristin Traylor
I said SUN. It’s the sensitive, useful, necessary… Essentially the data that you want to get, the relevant, the responsive. Legal data intelligence is a movement where we’re using these terms to say, like, we want to get rid of the ROT, essentially the junk, and get to the SUN, the really important documents, the things that are relevant and necessary. So that’s where that comes from. Sorry about using a bunch of acronyms.

Phil Favro
It’s okay. That’s great. So we’ve got a couple other questions. Brian asks whether the open… He says that he believes open source models are beginning to close ground and offer local control and fewer restrictions on usage for sensitive data types. And he says, “Do you see vendors or even corporate-end clients moving toward their own closed variants of open source weight models?” Anybody want to tackle that one?

Jim Sullivan
Oh, this one’s what I look for. So we’re analyzing new models every day. And frankly, most of these problems that we face can be solved generally easily with enAI when you use the highest quality tools. And the difficulty is: how can we get the best outcomes at a lower price point using other options? And what you’re talking about here, having open source models that give us more flexibility, more options, really opens the door to a lot more on-prem solutions where right now all of the best models are cloud-based. You have to send your data out somewhere. It has to leave the organization. And when you have open source, you can now run those locally; you can customize them in different ways. And ultimately, it gives us other options to get high-quality results at lower costs. And that’s essentially everything is moving in a direction where we’re getting more at a lower price point. So things that are cost-prohibitive today are not going to be cost-prohibitive in five years. And I think right now there are very, very few problems that we can’t solve with large language models if we had unlimited budgets, if I could use the highest-end model for every single thing, and I could run multiple passes and do a deep evaluation. But now we’re going to have the ability to do more things in the future because of this, and it’s really cool, and it’s going to give us more flexibility.

Phil Favro
Thanks, Jim. We got one last question. We need a 15-second response. It looks like Peter is asking, “From a policy perspective, are lawyers and law firms and the law school going to be pushing back because it sure sounds like generative AI is going to reduce all the need for lawyers?”

Jim Sullivan
Yes.

Phil Favro
Maybe just your five-second response on that, Cristin and Esther, if you choose.

Cristin Traylor
We still need lawyers, but now we can get back to doing what matters. Actually, lawyering, helping to advise our clients and, again, put the strategy together for the case and not worry about reviewing documents. I think we definitely need lawyers. It’s just that high-value legal judgment is really what we can go back to.

Esther Birnbaum
Yeah. And one final comment is we’re seeing law firms either push back hard because they’re scared or embrace it fully. And I recommend that both law students and lawyers learn to embrace the technology because your practice isn’t going to be extinct unless you don’t.

Phil Favro
Okay. So we got a shout-out: don’t be a dinosaur. Embrace the technology, use it, press forward with it, and understand how it’s going to enhance your practice and benefit your clients. Thanks, Jim, Aleida, Cristin, and Esther. Mary Mack, over to you.

Mary Mack
Thanks, Phil. And thank you all again for joining today’s HaystackID webcast. Thank you to our panelists for sharing their expertise. And for the Schulte case, Phil has provided his email. It’s [email protected], and he has a PDF for you. Before closing, please mark your calendars for HaystackID’s next webinar, “From Breach to Legal Crisis: The Hours that Define Your Exposure.” It’s happening on August 18th at 12:00 PM Eastern. And you can find that registration link in today’s resources, and we hope to see you there. And on behalf of EDRM, sincere appreciation is extended for your participation today, and wishing everyone a productive day. Thank you.

Esther Birnbaum
Thank you.

---

Expert Panelists

+ Philip Favro (Moderator)
Founder of Favro Law PLLC

Philip Favro is the founder and president of Favro Law PLLC, where he counsels clients on ESI, AI, and discovery issues and serves as a special master, mediator, and expert witness. Phil is nationally recognized for his expertise on ESI, discovery, and information governance, with courts acknowledging his credentials. See, e.g., Oakley v. MSG Networks, Inc., No. 17-CV-6903 (RJS), 2025 WL 2061665 (S.D.N.Y. July 23, 2025). This background makes Phil particularly well-suited to counsel clients and advise courts on information-related issues. As a special master, Phil is acclaimed for his collaborative approach, working with parties to find stipulated solutions to complex issues. For disputes that require adjudication, he is renowned for the clarity and vigor of his written dispositions.

+ Aleida Gonzalez
Global Advisory Managing Director, HaystackID

An accomplished attorney and military intelligence officer, Aleida Gonzalez brings a rare combination of legal expertise, national security expertise, and strategic advisory capability that uniquely positions her to lead in the field of AI governance. With nearly a decade of service as a prosecutor and extensive military experience, Aleida has operated at the intersection of law, policy, and security—advising senior leaders, managing complex investigations, and shaping outcomes in high-stakes environments. Building on her AI Governance certification and prior service as a prosecutor, Aleida’s professional development reflects decades of deliberate preparation for managing high-risk, high-consequence systems where accountability, policy, and operational disciplines intersect.

+ Jim Sullivan
Chief Executive Officer, eDiscovery AI

Jim Sullivan is the Founder and Chief Executive Officer of eDiscovery AI, a data intelligence company pioneering the next generation of AI-powered legal technology solutions. As CEO, Jim leads the company’s vision to transform the legal industry’s approach to litigation, investigations, information governance, and privacy response. Under his leadership, eDiscovery AI is setting new standards for accuracy, efficiency, compliance, and cost-effectiveness through advanced artificial intelligence tools that simplify complex data challenges. A licensed attorney and recognized authority in legal technology, Jim has spent nearly two decades at the forefront of integrating AI into legal workflows. He has consulted on thousands of predictive coding and analytics projects, guiding law firms and corporations through some of the most complex and large-scale matters in the industry. Jim’s deep expertise has made him a frequent conference and webinar speaker, where he shares practical insights on leveraging AI to improve productivity, defensibility, and outcomes in eDiscovery. Jim is also the author of The Book on AI Doc Review: A Simple Guide to Understanding the Use of AI in eDiscovery and co-author of The Book on Predictive Coding: A Simple Guide to Understanding Predictive Coding in e-Discovery. These works have become accessible resources for legal professionals seeking to understand and apply AI in practice. Known for his ability to demystify complex technologies, Jim is passionate about helping legal professionals embrace AI to future-proof their work and enhance the integrity of legal proceedings.

+ Cristin Traylor
Senior Director, AI Transformation and Law Firm Strategy, Relativity

Cristin Traylor is Senior Director of AI Transformation & Law Firm Strategy at Relativity, where she works closely with law firms, corporations, government entities, and partner organizations to align AI-driven legal data intelligence with organizational strategy, client service, and risk management. She brings a practitioner’s perspective to this work, drawing on her background as a former litigator and Discovery Counsel at McGuireWoods, where she handled complex Government Investigations & White Collar Litigation and Business & Securities matters, and led firmwide discovery strategy. Traylor advises the legal industry on how to responsibly and defensibly deploy AI across e-discovery and other high-stakes legal workflows, helping organizations enhance efficiency, consistency, and client value. A frequent author and speaker on generative AI and legal data intelligence, she also plays an active leadership role across the industry. Her service includes the Steering Committee of The Sedona Conference Working Group 1, liaison to Working Group 13 on Artificial Intelligence and the Law, the board of Richmond Women in eDiscovery and membership on the EDRM Global Advisory Council. She is a Relativity Master, Legal Data Intelligence Ambassador, and a member of the Virginia and District of Columbia Bar Associations.

---

About HaystackID®

HaystackID® solves complex data challenges related to legal, compliance, regulatory, and cyber requirements. Core offerings include Global Advisory, Cybersecurity, Core Intelligence AI™, and ReviewRight® Global Managed Review, supported by its unified CoreFlex™ service interface and eDiscovery AI™ technology. Recognized globally by industry leaders, including Chambers, Gartner, IDC, and Legaltech News, HaystackID helps corporations and legal practices manage data gravity, where information demands action, and workflow gravity, where critical requirements demand coordinated expertise, delivering innovative solutions with a continual focus on security, privacy, and integrity. Learn more at HaystackID.com.

Assisted by GAI and LLM technologies.

SOURCE: HaystackID