[EDRM Workshop Transcript] Balancing Ethics and Efficiency with GenAI in Legal 

Mary Bennett

Thank you. And David Horrigan, nice to have you here as well. David is the Discovery Counsel and Legal Education Director at Relativity. So, David and I have worked together many a time—so great to be working here with you today. He is an attorney, award-winning journalist, law school guest lecturer, and former eDiscovery industry analyst. He has served as counsel at the Entertainment Software Association, a reporter and assistant editor at the National Law Journal,and he’s been an analyst and counsel at 451 Research. David’s the author and co-author of law review articles as well asthe annual Data Discovery Legal Year in Review. He’s a frequent contributor to Legaltech News and was first runner-up for Best Legal Analysis in the LexBlog Excellence Awards. So, David, great to have you here as well.

David Horrigan

Mary, great to be back with you, just like old times.

Mary Bennett

Just like old times. And Anya, rivaling Dave for his library—I think they’re both pretty stacked here—but I work with Anya at HaystackID. She’s the VP of Cyber Incident Response and Advanced Technologies Group and has been with Haystack for 18 years—sorry, has been in the legal industry, sorry, Anya, for 18 years as a licensed attorney, 15 of which have been in eDiscovery, where she focuses on data mining, complex integrated workflows, and document review. In her role at HaystackID, Anya works on developing and implementing the strategic direction of our Cyber Incident Response team. She’s one of our industry’s leading experts on data breach incident response, notification, and reporting, and she has a very strong understanding of machine learning, custom object development, regular expressions manipulation, and other technical specialties. So, Anya, thrilled to have you here as well.

Anya Korolyov

Thank you so much, Mary.

Mary Bennett

Mary Mack. So, Mary is, as many of you know, the CEO and Chief Legal Technologist at EDRM, which is a project-based organization. She’s the former Executive Director of a certificate organization. Mary is very well known for her skills in relationship and community building, as well as her depth of eDiscovery knowledge. She’s frequently sought out by media for comments on industry issues and by conference organizers to participate in and moderate panels, lead workshops, or deliver keynotes. She’s the author of A Process of Illumination: The Practical Guide to Electronic Discovery, which we all consider one of the first popular books on eDiscovery. So, Mary Mack, thanks for joining.

Mary Mack

Thank you, Mary.

Mary Bennett

And I will go to Mr. Jim Sullivan, the founder of eDiscovery AI. Oh, I love the emojis. It is such a great use of those emojis. Thank you. He’s an accomplished attorney and a leading expert in legal technology. As the co-founder of eDiscovery AI, Jim is at the forefront of transforming how the legal industry uses advanced AI in document review. With more than two decades of experience, Jim has become a recognized authority on integrating AI into legal workflows and has played a key role in modernizing eDiscovery practices. Jim, it’s lovely to have you here.

Jim Sullivan

Always my pleasure. Thanks.

Mary Bennett

And Mary Bennett—as many of you or some of you may know me as Mary Rectoris—I’ll respond to either. I’m still getting used to the rebrand here, as you’ve heard me joke many times, though that joke will wear thin soon. I am the Director of Content Marketing at HaystackID. I’m really passionate about storytelling and using storytelling to educate our industry on pressing issues. I have about 10 years of content marketing experience. I was in the agency world for a couple of years before joining HaystackID. Before that, as I mentioned, I worked with David Horrigan at Relativity, where I was a Brand Programs Producer. I’m so excited to be back in the industry for the last year and to continue leading these workshops. So, we have a lot we want to get through today. As Holley shared at the beginning, there’s a Q&A feature. I will be tracking that alongside my fellow panelists as well. So, if you have questions, we can address them throughout or leave some time at the end. Kicking us off here, given that we are now in November—it’s crazy—it’s been a busy year. We’ve been discussing this since our first workshop, Jim, in March—or was it February? It’s been a whirlwind, but a lot has happened over the last few months, even the last few weeks, with GenAI. So, why don’t we start with you, Jim? What’s been the most significant update you’ve seen in the last few months that has impacted legal practices, and how are you seeing the industry respond to these changes?

Jim Sullivan

Yeah, I got to say that the general availability release of aiR, I felt like it was a really big step in the market towards really making AI review tools accessible to a much larger audience. So that’s something that, while we’ve been talking about AI and AI review for a really long time—well, maybe not a long time—this is really the first time that it’s been available to a majority of the users. So, we’re really excited about that and really excited for people to see how great that technology is.

Mary Bennett

Thank you, Jim. And that seems like a seamless transition to you, David Horrigan, on the topic of aiR. What have you been seeing?

David Horrigan

Well, first, let me state that I’m glad that Jim has seen and found aiR to be very beneficial. Obviously, we think it is, too, but I’m going to go outside the box and focus on an event that probably hasn’t caught much attention yet. I think that Dave and I are going to be discussing this during the conversation, and it’s an order by the Supreme Court of Delaware: its interim policy on the use of generative AI by judicial officers and court personnel. I think when AI first came out, there was a reaction, as there is to any new technology. Artificial intelligence has been around for a while, but on November 30, 2022, with the release of ChatGPT, the conversation and activity increased dramatically. This order from the Delaware Supreme Court came out last month, on October 21, and it governs the use of GenAI by judicial officers, judges, and court personnel. It’s significant because most of what came out before—and frankly, I think all of what came out before—was governing what litigants can do, what lawyers can do, and what legal teams or paralegals can do. This is significant because it focuses on the judiciary. It’s also significant because it takes a measured approach. When GenAI first came out,—and EDRM has covered this so well in its compilation of standing orders—there were standing orders that almost prohibited the use of AI. Some talked about disclosure, and certain courts even backtracked. This one, however, talks about judicial personnel’s use and is measured and balanced. So, it might not be grabbing big headlines yet, but I think it’s going to have a big impact because what judges and court personnel do affects all litigation. So, that’s my pick, Mary.

Mary Bennett

Thanks, David. And we’ll get into all of that great stuff later, but thank you very much. Dave Cohen, following up—any thoughts on what David said, or do you want to take a different angle on the biggest change you’ve seen as of late?

Dave Cohen

I agree with both Jim and David, but let me add something new, which is this: GenAI came into everyone’s attention about two years ago. We’re approaching the second anniversary of the release of ChatGPT version 3.5, which was on November 30, 2022—a date that will live in infamy! In those two years, a lot of time has been spent trying to incorporate it into new products and figuring out how it works. In the last few months, I’ve noticed that it’s now integrated into several products, like Relativity’s aiR for Review. But beyond that, I think everybody was cautious about starting to use these tools. They weren’t well known, and they weren’t accepted. Many corporations and law firms had rules prohibiting their lawyers from using AI until the dangers were identified and mitigated. What I’m seeing in the last few months is that the light has turned green. People are actually starting to use this in real cases, and it does some things very well. It’s great as a research starter and for summarizing long documents or groups of documents. So, in addition to the use cases already mentioned, we’re finding more, and adoption is growing rapidly.

Mary Bennett

Love that. Thank you, Dave. Anya Korolyov, what about you? What’s been the biggest change you’ve noticed in the last couple of months?

Anya Korolyov

I have to agree with Dave quite a bit. I think the acceptance of the fact that these tools are here and that people are starting to use them is significant. I also think the release of aiR, as Jim said, has been a big factor. At HaystackID, we’ve been using AI tools in incident response for a few years now, and even within that space, I’m seeing attorneys who were hesitant—attorneys who were very cautious and didn’t want to go down that road—start to accept that AI is here, that it’s going to be helpful, and that it’s going to be efficient. So, I think the last few months have really brought about a lot of acceptance and curiosity.

Mary Bennett

Yes. I was with my family the other day, and my sister was telling my dad, who’s an attorney and can barely use Zoom, that he could have ChatGPT on his phone. He was very interested. So, that seems to confirm what you’re saying, Anya. Why don’t we go to you, Mary Mack?

Mary Mack

Thanks, Mary. I think for me, the most significant thing is the adoption curve. It feels like it’s going up, up, up. At the same time, the speed of improvements in the technology—the underlying generative AI technology and the advancements being made on top of it—has been more rapid than I ever believed possible.

Mary Bennett

Awesome. And Mary Mack, staying with you—how do you see GenAI reshaping litigation practices? Dave Cohen mentioned there were some cautionary issues around it. What are some of the persistent challenges, and how do you weigh those against the benefits?

Mary Mack

The benefits are just incredible. GenAI can synthesize information in ways that used to be considered purely human qualities—things we hold very closely as attorneys. This technology moves that process forward so quickly. I think we’re going to see a lot of higher-level tasks automated in legal practice in addition to the lower-level ones. And because legal practice is all about dealing with information, I think every week we’ll see a new application.

David Horrigan

While Mary Mack is gathering her thoughts, I’ll jump in to expand on her point. We at Relativity just released a study in collaboration with IDC, and some of the findings really align with what’s being said here. One of the biggest takeaways is the level of trust in eDiscovery tasks when it comes to generative AI. For example, 89% of respondents said they’re comfortable using GenAI for document review—either very comfortable or somewhat comfortable. It’s a clear indication that confidence is growing, and we’re seeing GenAI integrated into more areas of practice.

Mary Bennett

I appreciate the insights, David. Mary Mack, are you back with us?

Mary Mack

Yes, sorry about that. To pick up where I left off, the challenges are real, especially around ethics and ensuring proper oversight. The benefits are clear, but the risks—such as hallucinations or issues with confidentiality—require vigilance. However, I think the speed at which this technology is evolving gives us the opportunity to continuously refine how we use it. We must prioritize transparency and validation in these workflows.

Mary Bennett

Thank you, Mary Mack. David, you’ve touched on ethical concerns briefly, but let’s dig deeper. When advising clients or industry peers, what do you tell them about integrating GenAI tools like ChatGPT into ESI protocols? How do you approach potential ethical violations, especially regarding confidentiality?

David Horrigan

Great question, Mary. And let me preface by emphasizing that this is not legal advice—just educational insights. One of the first ethical concerns with GenAI tools is confidentiality. Public models, such as ChatGPT, don’t guarantee the confidentiality of the data you input. Lawyers have a duty to protect client information, so using public models without proper safeguards can be a huge risk. A second major concern is hallucinations. As Dave Cohen mentioned earlier, we’ve seen cases like Mata v. Avianca, where attorneys cited fake cases generated by AI. These issues underscore the importance of verifying all outputs before relying on them. On the topic of disclosure, whether you need to disclose that you’ve used GenAI to your client depends on the circumstances. ABA Formal Opinion 512 highlights that attorneys must maintain transparency and competence, which includes understanding the technology they use.

Dave Cohen

I’d like to add to that. In addition to confidentiality and hallucinations, we need to be cautious about how opposing parties might use GenAI. For example, if you produce documents for them, they might upload those into GenAI tools to generate summaries. You can mitigate this risk by including specific provisions in your ESI protocols or protective orders to limit how these tools can be used with your client’s data.

Mary Bennett

Thank you, Dave. Anya Korolyov, what are your thoughts on ethical concerns and best practices for integrating GenAI into workflows?

Anya Korolyov

I completely agree with Dave and David. Confidentiality is a huge issue, especially when you’re dealing with sensitive data in incident response or breach notification workflows. At HaystackID, we’ve been very intentional about vetting the tools we use to ensure they meet our confidentiality standards. Another point I’d like to emphasize is defensibility. If you’re using GenAI to analyze or summarize documents, you need to document your process. How did you use the tool? What outputs were generated? And how did you validate those outputs? Being transparent and thorough will go a long way in mitigating risks.

Mary Bennett

Great points, Anya. Let’s shift gears a bit. Jim Sullivan, you’ve been closely monitoring advancements in GenAI tools like Relativity’s aiR. How do you see this technology evolving, particularly in comparison to traditional TAR (technology-assisted review) workflows?

Jim Sullivan

Honestly, it’s not even a comparison. With traditional TAR, especially TAR 1.0, you’re aiming for around 80% recall and maybe 60% precision on a good day. With GenAI, we’re consistently seeing over 90% recall and 90% precision. It’s night and day. The accuracy, the ability to handle foreign language documents, and the contextual understanding GenAI brings are game-changers. What’s more exciting is how GenAI fills gaps that TAR never could. For example, TAR struggles with non-text documents or understanding context across multiple languages. GenAI handles these challenges seamlessly. We’ve conducted head-to-head comparisons, and the results speak for themselves—GenAI outperforms human review teams in recall and matches or exceeds them in precision.

Mary Bennett

Thank you, Jim. Anya, do you agree with Jim’s assessment? Are you seeing similar trends at HaystackID?

Anya Korolyov

Absolutely. I think Jim hit the nail on the head. GenAI is not only more accurate but also significantly more efficient, especially for large, complex matters with multiple issues. The time savings alone make it a compelling option, but you still need a subject matter expert to guide the process. Prompts and training are critical to ensuring the tool delivers accurate results.

Mary Bennett

Thanks, Anya. Dave Cohen, let’s expand on that. How do you see TAR, GenAI, and traditional methodologies like search terms coexisting in workflows? Are we moving toward a unified approach, or do you see specific use cases for each?

Dave Cohen

I think there’s still a place for all three. While GenAI is revolutionary, traditional TAR and search terms aren’t going away anytime soon. For example, in cases with millions of documents, using search terms to narrow the population is often the most practical and cost-effective first step. From there, you can bring in GenAI or TAR to refine the results. What sets GenAI apart is its ability to start without extensive training, unlike TAR, which requires iterative training and feedback. However, TAR’s continuous active learning is still an advantage in some workflows because it adapts to reviewer feedback in real-time. Ultimately, the best approach depends on the specific case and client needs.

Mary Bennett

That’s a great perspective. Jim, does the rise of GenAI change how you think about the role of search terms in eDiscovery? Are you seeing pushback against their use in favor of more advanced methods?

Jim Sullivan

That’s a fascinating question. I think search terms will always have their place, especially as a starting point to reduce the document population. But when you look at what GenAI can do—contextual understanding, multilingual capabilities, and identifying key documents—you start to question whether search terms alone are enough. In some cases, I’ve seen clients push back against search terms, arguing that they miss too much relevant data. However, there are practical limitations, such as cost, that often make search terms a necessary step. The key is finding the right balance and educating clients on the strengths and limitations of each approach.

Mary Bennett

Thank you, Jim. Anya, let’s pivot to metrics and budgets. At Relativity Fest, you spoke about how GenAI is changing how we think about metrics and estimates. Can you share your insights on how GenAI impacts cost predictability and efficiency?

Anya Korolyov

Absolutely. Metrics and budget estimates are crucial in eDiscovery, and GenAI is shaking up how we approach both. Traditionally, we’ve relied on historical data to predict costs and timelines, but GenAI introduces new efficiencies that aren’t fully reflected in those models yet. For example, the ability to achieve higher precision and recall with less time spent on manual review significantly reduces costs. However, the pricing structure for GenAI—whether per document, per prompt or another model—can make budgeting a bit tricky. Over time, as we gather more data on how GenAI performs across different types of matters, we’ll be able to refine our metrics and provide more accurate estimates. For now, it’s important to approach each case individually and stay transparent with clients about potential costs and savings.

Mary Bennett

Thank you, Anya. David Horrigan, you mentioned earlier the IDC study commissioned by Relativity. Can you share more highlights from that research, particularly around adoption and trust in GenAI?

David Horrigan

Of course, Mary. The IDC study is a comprehensive look at how GenAI is being adopted across the legal industry. One key finding is that 89% of respondents are comfortable using GenAI for document review, which leads all other eDiscovery tasks. The automation of basic tasks was cited as the number one reason for adopting GenAI, and cost savings closely follow it. Another interesting point is the growing trust in GenAI’s ability to handle complex tasks. While some concerns remain around hallucinations and confidentiality, the data suggests that these issues are becoming less of a barrier as organizations gain more experience with the technology. It’s exciting to see how quickly adoption is growing, and I think we’re just scratching the surface of what’s possible.

Mary Bennett

Thank you, David. Let’s address a question from the audience about the discoverability of GenAI prompts. Dave Cohen, are GenAI prompts discoverable, and should they be retained as part of litigation records? What’s your take?

Dave Cohen

Great question, and the answer is—it depends. If the prompts are created by an attorney in anticipation of litigation, they could be considered work product and may be protected from discovery. However, if a client or non-attorney creates the prompts outside the context of litigation, they may not have the same protection. Another consideration is documenting the prompts and outputs for defensibility. If you’re using GenAI to generate summaries or analyze data, you need to be able to explain and validate how the tool was used. Transparency and thorough documentation are critical to ensuring that the use of GenAI withstands scrutiny.

Mary Bennett

Thank you, Dave. Mary Mack, any final thoughts on how organizations can stay ahead of the curve with GenAI while addressing these emerging challenges?

Mary Mack

Yes, I think it comes down to collaboration and education. Organizations need to invest in training their teams to understand and use these tools effectively. They also need to create clear policies and protocols for how GenAI is used, ensuring that ethical considerations are addressed upfront. Lastly, staying engaged with the broader legal tech community through events like this one is essential to staying informed and prepared.

Mary Bennett

Wonderful insights, Mary. Thank you. And thank you to all our panelists—Jim, David Horrigan, Dave Cohen, Anya, and Mary Mack—for an incredible discussion. I’d also like to thank our audience for joining us today. As mentioned earlier, HaystackID has an upcoming webcast on December 11th focused on creating a unified strategy for removing data silos. Visit our website to register.

Mary Mack

Thank you, Mary, and thank you to HaystackID for partnering with EDRM on today’s workshop. This concludes our session. We look forward to seeing you at our next event on November 14th, covering eDiscovery trends and predictions for 2024 and 2025. Have a great day, everyone!

---

Expert Panelists

+ David R. Cohen
Partner, Reed Smith, LLP

David Cohen is the chair of Reed Smith’s Records & E-Discovery (RED) Group and a member of the Emerging Technologies group. A Harvard Law graduate with more than 35 years of commercial litigation experience, Cohen serves as eDiscovery counsel and information governance counsel to some of the top companies in the world. He also represents clients in complex litigation matters and counsels companies of all sizes on information governance and litigation readiness issues.

Cohen has been recognized individually by Chambers Global, Chambers USA, Super Lawyers, Best Lawyers, and Who’s Who Legal as a top eDiscovery lawyer and litigator. He has also received a Law 2.0 “Outstanding Leadership” award in 2022 and the Legal Intelligencer Pennsylvania “Innovator of the Year” award in 2023. In addition to individual recognition, the 70+ lawyer RED Group that he leads has been recognized by Chambers and Legal 500 as a leading eDiscovery practice.

---

+ David Horrigan

Discovery Counsel & Legal Education Director, Relativity

David Horrigan is Relativity’s discovery counsel and legal education director. An attorney, award-winning journalist, law school guest lecturer, and former e-discovery industry analyst, he has served as counsel at the Entertainment Software Association, reporter and assistant editor at The National Law Journal, and analyst and counsel at 451 Research. The author and co-author of law review articles as well as the annual Data Discovery Legal Year in Review, David is a frequent contributor to Legaltech News, and he was First Runner-Up for Best Legal Analysis in the LexBlog Excellence Awards. His articles have appeared also in The American Lawyer, Corporate Counsel, The New York Law Journal, Texas Lawyer, The Washington Examiner, and others, and he has been cited by media, including American Public Media’s Marketplace, TechRepublic, and The Wall Street Journal. Horrigan serves on the Global Advisory Board of ACEDS, the Planning Committee of the University of Florida E-Discovery Conference, and the Resource Board of the National Association of Women Judges. He is licensed to practice law in the District of Columbia, and he is an IAPP Certified Information Privacy Professional/US.

---

+ Anya Korolyov
Vice President, Cyber Incident Response and Advanced Technologies Group, HaystackID 

---

+ Jim Sullivan
Founder, eDiscovery AI

Jim Sullivan is the founder of eDiscovery AI. With over a decade of experience consulting on predictive coding and analytics use, he has a passion for analytics and calculates recall and precision metrics in his sleep. His life is a non-stop adventure of trying to find better ways to solve problems.

---

+ Mary Bennett [Moderator]
Director, Content Marketing, HaystackID; Senior Director, Content and Community Initiatives, EDRM

Mary Bennett, HaystackID’s Director of Content Marketing, focuses on the power of storytelling to educate the legal technology industry on pressing issues impacting practitioners. With nearly 10 years of content marketing experience, Bennett joined HaystackID after working at an agency to help B2B tech startups grow their marketing engines through content that drove audiences through the marketing funnel.

Before her agency experience, Bennett worked at Chicago-based Relativity as a Senior Producer on the Brand Programs team. She was a founding member, host, and producer of Relativity’s Stellar Women program and producer of the company’s documentary series, On the Merits. In her role, Bennett crafted and socialized important stories that elevated the eDiscovery community and illustrated technology’s potential to make a substantial impact.