[EDRM Workshop Transcript] From Summation to AI Tokens: A Journey Through eDiscovery’s Most Transformative Shifts

Mary Mack

Hello, and welcome to our second quarter EDRM workshop. I’m Mary Mack, EDRM’s CEO and Chief Legal Technologist. Today’s workshop, which is a collaboration with EDRM’s trusted partner HaystackID, is called “From Summation to AI Tokens: A Journey Through eDiscovery’s Most Transformative Shifts.” Our workshop experts are the Honorable Judge Young Kim, Doug Austin, Adam Rubinger, Dennis Kiker, Caroline Sweeney, and myself. Our wonderful moderator and organizer is Mary Bennett, Senior Director of Content Marketing at HaystackID and Senior Director of Content and Community Initiatives at EDRM. And with us, we have Holley Robinson, EDRM’s Marketing Operations Manager, to tell us about some resources we have available to you. Holley. 

Holley Robinson

Thank you, Mary. We are loving ON24 with so many more ways for you, our audience, to engage. If you check out the console to the right of your screen, you’ll see a question mark icon. This is where you can type in your questions for today’s faculty, and we highly encourage you to do so. You’ll also see the paperclip icon for related content. If you click on it, you will see today’s resources carefully curated just for you, including the description of the ON24 engagement tools for you to download and better engage with the platform. There is also a link to learn more about HaystackID’s exceptional services. Be sure to take a few minutes to share your insights on the eDiscovery Business Confidence Survey 1H2025. It only takes a few minutes. There’s also a link to schedule a meeting with the experts at EDRM’s trusted partner, HaystackID. Next, on the EDRM webinar channel on April 23rd, all your data will be in one place, streamlining eDiscovery. The speaker bios can be clicked and pop up to learn more about today’s faculty. Just like on Zoom, you can click the smiley face emoji to react to the webinar, and we highly encourage you to do so. We’re having so much fun with the emojis on this platform. And finally, the icon for social media. You’ll be able to follow EDRM and HaystackID on LinkedIn. Back to you, Mary. 

Mary Mack

Thanks, Holley. And Mary Bennett is wearing the two hats today, and a title upgrade. She is the Senior Director of Content Marketing for HaystackID. And as I said earlier, she is the Senior Director of Content and Community Initiatives for EDRM. For five years, she was Brand Producer at one of our great trusted partners, Relativity, where she was a Founding Member, Host, Producer, and all-around organizer of Relativity’s Stellar Women Program. She was the Producer of the company’s documentary series “On the Merits.” Mary recently received JD Supra Readers’ Choice, Top Author Accolades. She has a bachelor’s from the University of Iowa. She is a storyteller, a community builder, a content strategist, and an eDiscovery marketer. Without further ado, the reins are yours, Mary.

Mary Bennett

Thank you, Mary. And welcome, everyone. We are so excited about today’s workshop. As Holley and Mary noted, we encourage you to use the ON24 platform today. My biggest note will be that we want these workshops to be interactive, especially today’s topic. We’re going through the phases of eDiscovery. I’m sure many of you have been in the industry and have your own stories or questions to share. So, if you would like, there is a Q&A panel where you can type your questions and thoughts, and I will be monitoring that throughout. We’ll also be recording today’s presentation. So, if someone had a really great insight or I placed a very well-executed joke that you want to relive, you can do that after the presentation as well. We have a lot to get through today, but not a ton of time. So, I’m going to jump into our amazing speakers, starting with Doug Austin, the Editor at eDiscovery Today. Doug is an established eDiscovery thought leader with more than 30 years of experience providing eDiscovery best practices, legal technology, consulting, and technical project management services to numerous commercial and government clients. Doug has published a daily blog since 2010, which is impressive coming from a fellow writer. He has also written numerous articles and white papers. He has received the JD Supra Readers’ Choice Award as the Top eDiscovery Author for 2017 and 2018 and a JD Supra Readers’ Choice Award as a Top Cybersecurity Author for 2019. Doug, we are thrilled to have you. Thanks for joining today.

Doug Austin

Good to be here.

Mary Bennett

We then have Dennis, a Senior Attorney at DLA Piper. Dennis is a litigation attorney with extensive eDiscovery experience. He focuses on guiding clients and litigation teams on the strategy and implementation of defensible practices associated with identifying, preserving, analyzing, reviewing, and producing documents and data designed to help our clients realize better results in less time and at a lower cost. He is among the firm leaders in piloting and implementing AI and other advanced technologies to improve outcomes and lower costs. So, AI, as with all conversations lately, will be a big part of today’s conversation. So, Dennis, we are looking forward to hearing how the firm and you are using AI and exploring those tools.

Dennis Kiker

Thanks so much.

Mary Bennett

Then, we have a fellow Illinois person, Judge Kim, and me. He is the Magistrate Judge of the Northern District of Illinois. So, welcome, Judge Kim. He was born in South Korea, and his family immigrated to the U.S. when he was 11 years old. He is a graduate of the University of Illinois at Urbana-Champaign and Loyola Law School in Chicago. He began his legal career as an Assistant County Cook Public Defender in 1991. From 1993 to ’95, he clerked for the District Charles R. Norgle of the Northern District of Illinois. Currently, as a federal judge, he particularly enjoys presiding over citizenship naturalization ceremonies in the ceremonial room where he himself was naturalized as a U.S. citizen in 1986. Judge Kim, thank you for joining. Mary Mack probably needs no introduction, but I will do one nevertheless. She is the CEO and Chief Legal Technologist at the EDRM, a project-based organization. She’s the former executive director of a certification organization. We all know Mary for her skills in relationship and community building and her depth of eDiscovery knowledge. She’s frequently sought out by the media for comments on industry issues and by conference organizers to participate, moderate a panel, lead a workshop, or deliver a keynote. She’s the author of “A Process of Illumination: The Practical Guide to Electronic Discovery,” which is considered by many to be the first popular book on eDiscovery. So, Mary, as always, it’s lovely to have you.

Mary Mack

Thanks, Mary.

Mary Bennett

And then Adam, who is my colleague at HaystackID. He is the Chief Client Experience Officer. In this role, Adam works directly with our corporate clients on their eDiscovery team as the executive sponsor. He provides guidance, consultation, and advice to teams in all areas of the process, from collections to productions. He’s very much involved in most aspects of the portfolio and works at all levels of the organization. Adam, I am excited to have you here.

Adam Rubinger

Thank you.

Mary Bennett

We have Caroline Sweeney, who is the Director of Knowledge Management and Innovation at Dorsey & Whitney. She is responsible for delivering their eDiscovery services, including LegalMine Managed Review services, litigation technology support, and trial technology support. Caroline is a member of Dorsey’s Electronic Discovery Practice Group and the Cybersecurity, Privacy, and Social Media Practice Group. She has extensive experience consulting with attorneys and clients regarding eDiscovery, including the identification, preservation, collections, processing, review, and production of ESI. Caroline, we are thrilled to have you here as well. Mary introduced me, and I want to say that I am excited to be here and to moderate today’s panel. With that, why don’t we dig into the conversation? Adam, I’m going to invite you to kick us off. Looking back on your career in eDiscovery, what moment from your early days made you realize this industry is changing so quickly?

Adam Rubinger

Well, thanks, Mary, and super excited to be here and talking with everybody. And thanks to everybody who joined us. I’ve been in this industry since about 1996, when it was paper before there was an E in front of discovery. I had the pleasure of one of my first jobs in this space, working with Mary Mack. And so, we’ve gone back a long way, the two of us, and I’ve known everyone on this panel for quite some time. So, I’m excited to do this. The big aha moment came when we started looking at predictive coding. In the early days, it was basically just digitizing paper and doing linear review like we had always done. Along came predictive coding, which opened everyone’s eyes to, “Ooh, we can use technology even more efficiently, faster, maybe not cheaper,” but in the long run, it became cheaper. That, for me, was when I was able to see that the space had changed and that we would see an acceleration of technologies. We were already seeing data processing speeds increase pretty exponentially over the first, let’s say, 10 years before predictive coding became at least accepted and, in some cases, mainstream. That worked for me; it was the aha moment when things would evolve quickly.

Mary Bennett

Thank you, Adam. And Judge Kim, coming from a different side of the bench, what was your aha moment when you realized this industry is changing so rapidly?

Judge Young Kim

I’m not sure I had an aha moment, but I think I had an oh-crap moment. So, when I joined the bench in 2010, the first thing that I noticed was that in every single civil case, a standard order was entered, basically telling the parties that they needed to follow the Seventh Circuit’s trial eDiscovery protocol. Of course, when I’m looking at this, I’m thinking to myself, “What is eDiscovery?” Because until then, I was not familiar with eDiscovery. Everything was done in documents and PDF format, and I didn’t have many eDiscovery issues. Just before I took the bench in 2010 and for nine and a half years, there was no issue regarding eDiscovery. It was an oh-crap moment for me to learn as fast as possible about eDiscovery issues when I took the bench.

Mary Bennett

Thank you. And Caroline, either an aha moment or an oh-crap moment, what was one of those for you?

Caroline Sweeney

I guess it was kind of a combination. Like Adam, I’ve been in this industry for quite a while, and I’ve seen a lot of transitions. I remember when we first started automating deposition transcripts. I remember manual paper and was a project manager of manual coding of paper documents, hosting that on a mainframe computer, and dialing into the mainframe to search the database. I remember the introduction of PC-based tools like summation. At that time, there were Inmagic, BRS Search, and later Concordance. But when it comes to eDiscovery, it was probably very early on when it became a thing. This was the early 2000s. We had a massive class action matter. We had hundreds and hundreds of boxes. And I remember asking the attorneys, “Well, what about email?” Because email was a thing by then. And I remember them saying, “Well, we’ve agreed not to raise it with the other side, and we’re hoping they don’t raise it with them.” And I remember thinking that I felt relief because, like, “Oh my God, thank God. We didn’t know a lot because it was so early on.” There was a lot to learn, and that would have been our first big experience with email. And if you recall, it was so expensive at that time, and the vendors that you dealt with back then, probably a lot of them don’t even exist anymore. And we were talking about restoring email from backup tapes and all of that. But I also remember thinking, “I wonder how long that’s going to be the case that we’re just not going to talk about email and hope it doesn’t rear its head.” Sure enough, it wasn’t too long before another large matter came in, and then we suddenly were dealing with backup tape restoration for email and reviewing that. I think what the aha moment was in many ways was up until that time; it was when we only really automated or applied litigation technology tools to that large kind of bet-your-company type of litigation. And now suddenly it was, “Hey, we have electronic data.” And eventually, it was in every case, and there’s hardly any paper now. But our group, the whole litigation technology group, became a different type of resource and valued in a different way. And that’s when I knew this was really going to be so early on. I thought this was really going to be transformative, and we have the opportunity to really demonstrate how technology can help in these situations or in litigation.

Mary Bennett

I know today, we’re going to be talking about the different evolutions and how you’ve had to lead your team, transform their skill set, and show their values. I think that’s interesting as well. Dennis, why don’t you share your big aha moment, please? 

Dennis Kiker

It’s funny. I was listening to Caroline, and it reminded me of a meeting we had with a big client and all the litigation counsel. We’re one of the national counsel for that client, and they had a meeting to talk about eDiscovery. They announced that the email was not discoverable. That was their determination to deal with email at the time. So, it has kind of changed since then. I mean, there have been so many. One thing that’s fun about this industry is that you can just look back, and there’s always an aha or an oh-crap moment at some point in time. But I think the one from the early days that struck me was when we went to web-based review platforms. Up until then, I was a summation shop. Our firm was a summation shop, and we would invite people in, sit around in the conference room, and code documents. And when web-based platforms came out, it just opened up an amazing way to expand the resources to begin to staff those kinds of projects to work more collaboratively across jurisdictions and so forth. So, that was a big, big moment. Mary was pivotal back at Fios with one of the first platforms, but that was big for me. It was a huge change in how we approach this kind of work.

Mary Bennett

Thank you, Dennis. Mary, it seemed like that was a good transition for you and the biggest thing you’ve seen, an aha or oh-crap moment.

Mary Mack

Well, mine was a bit of both. And that was when the federal rules were being introduced and socialized. There’s a gentleman that Carl Liggio introduced me to, Al Driver. He had a paper called The Metropolitan Corporate Counsel. Directed it in-house. It was all about the changes in the federal rules for 2006. I came into the industry as a technologist more than a legal person, but Al said, “I want you to read these rules and tell me what you’re thinking. I’ll do an interview with you.” So, I read the rules, and I was absolutely horrified. It was like, “The penalties for not complying and the idea that the rules were supposed to reduce eDiscovery.” As I read them, that wasn’t possible. And I told that to Al, and Al told me, “Don’t tell anybody.” He’s like, “Everybody thinks this is going to reduce eDiscovery.” And I said, “It is not.” Tom O’Connor was one of the only people willing to raise his voice then. And we’ve seen the evolution, but I thought, “Oh my God, the floodgates are so wide open right now.”

Mary Bennett

Thank you, Mary. And Doug, what about you? What was your biggest moment from the early days of our industry?

Doug Austin

Thanks, Mary, and hi, everybody. I will return to my first litigation support project because there wasn’t any discovery back then. We developed what we called a CAR system, which was computer-assisted retrieval of microfilm. And it was pretty cool. You’d do a search in the document database, pull doc IDs, and feed them into the program, which was on a PC attached to a microfilm reader printer. The program would tell the operator the first cartridge to load in the spool. When they hit the go button, it would automatically advance to the first page to print and start printing pages. There were markings at the bottom of each page to enable the program to locate and print the correct pages. It was pretty cool when it worked, but it often didn’t, which made me a very busy support person. Having worked a couple of summers in college in a file warehouse for a large insurance company, I could see that the days of having documents on paper were going to be a thing of the past one day. So, that was an aha moment, and hey, paper will not be the thing forever. And that’s a precursor to true eDiscovery, but it was certainly the first moment for me to see where we could be heading.

Mary Bennett

Definitely, the other day, I had to drive half an hour to my parents’ house to print something because I’d never needed to print anything. So, it is similar but definitely less important. I want to make a note to those who have just joined us: Welcome. And if you have anything you would like to share about your aha moment or questions for these amazing speakers, please use the Q&A, and I’ll be monitoring that. Doug, staying with you here, you’ve written about eDiscovery as we’ve discussed. What has surprised you the most about the industry adoption, or lack thereof, or dare I say, resistance to some really big changes like cloud migration or others?

Doug Austin

Early in my career, my boss told me to read the book “Crossing the Chasm.” The book segmented the technology market into different groups, including innovators, early adopters, and laggards. He told me many lawyers fall into the laggers category when it comes to adopting technology. And guess what? He was right. I was prepared for that. What I wasn’t prepared for is how lawyers personally used cloud-based solutions like online banking, or they used systems that used machine learning, like Spotify or Netflix, which, of course, as everybody knows, suggest songs or shows based on your viewing habits. But those same lawyers wouldn’t consider cloud-based solutions or the use of TAR-free discovery. And that’s been a disconnect that I’ve never fully understood. The legal profession has always been more risk-averse than other professions. Sometimes, lawyers are resistant to technology solutions that can make them more efficient, but for some lawyers, efficiency means billing less. Unfortunately, I think we’re battling that with every new technology that comes along. So, I continue to be surprised by things like the cloud and TAR, and we’ll see what happens with generative AI (GenAI) and how quickly lawyers embrace the new technologies. But we’ll see what happens.

Mary Bennett

We have quite a few lawyers on this call. Would anybody like to share their thoughts, even if they are not themselves? I know you all are innovators in the space, but do you have any thoughts on that? Do you think adoption is getting quicker from maybe TAR to GenAI? I’m open to the panel to respond or give some thoughts.

Dennis Kiker

This is Dennis. I think that there are… and some parts share Doug’s perception. I have to defend the lawyers, though. It’s not all about billable hours. Risk is a huge thing for lawyers. And so, not understanding, and that’s one of the challenges with new technology: It’s really easy to understand what a human is doing when they look at a document and make a decision. It becomes more difficult when you begin introducing algorithms or, heaven forbid, AI models, large language models that are making decisions that you don’t understand how that’s done. And you realize that at some point, you may be called upon to defend that process before a court and explain to Judge Kim exactly how you came to the results you did. That’s a frightening prospect for a lawyer who doesn’t understand the details. I can empathize with that, but we’re in an interesting inflection. We’re going to talk about this at some point with AI. In some ways, it is easier for lawyers to accept AI because of its general adoption. Everybody is using ChatGPT, Gemini, or Rufus to shop on Amazon. And so, people are in their normal lives getting more comfortable with the idea of AI, which I think, in some ways, makes it easier to digest and appreciate than TAR. This algorithm uses support vector models to calculate, and that’s hard to understand, digest, and communicate. But in some ways, this AI pivot we’re making now is easier for lawyers to digest. Law firms, including ours, are leaning toward adopting AI for several different functions, from legal research to contract development to several other areas. So, I completely understand most lawyers’ reservations and the risk-based aversion to change, but I think it may be accelerating and changing right now.

Mary Bennett

I think law schools are integrating it into the classroom. Is that correct, Dennis?

Dennis Kiker

It is true. It’s been a while since I’ve been to law school. And I don’t know, I can’t tell you what they’re doing at Michigan. But I know that locally, at the University of Richmond Law School, there are eDiscovery classes. They’re introducing lots of new technologies. Lawyers are coming out of school and are much more equipped to deal with the issues that are going on today. At least they’re conversant in those things. But it’s also, you make a good point that the lawyers coming out of law school are pretty technologically adept. They grew up with technology in their hands, really powerful technology in their hands. And so, it’s much easier for them to dive into and embrace the technologies that we use because, for them, it’s just part of what they’ve always done their entire lives, whereas we older folks had to learn this stuff as we went and get integrated into it at a different. I mean, you were asking about the big aha moments. I immediately thought of the bait stamp machine. Remember the stamps, actual ink stamps, and it would increment one by one, so you could do pages without stickers? I thought that was amazing. So, it’s much easier for people coming out of law school and entering the practice of law now to expect to rely on technology immediately.

Mary Bennett

Thank you. And this lends itself nicely, Judge Kim, to our next question, especially from where you’re sitting. What is your assessment of attorneys who need to explain those technical concepts to you? How has that changed over time? We’d love to hear your thoughts on that. I think you’re muted, Judge Kim.

Judge Young Kim

Sorry about that. The need has not changed. There’s always been a need from the bench’s perspective when dealing with technical issues, and no one is an expert in the area. Even a very tech-savvy attorney who is able to explain the process of eDiscovery collection, processing, and analyzing may find it difficult for the bench to understand. Also, when an attorney tries to explain, the opposing side always takes a negative view of that explanation. There’s this level of distrust there. And so, you get into an argument over whether what the attorney is saying is, in fact, accurate or reliable. For me, whenever I have a technical concept that I need to understand because of the gravity of the situation or whatever the case may be, I always order the parties to have the experts available by phone so that I can ask questions of that expert and not necessarily a declaration or an affidavit from the expert. I prefer to have a conversation with the expert on the record and also allow the opposing attorney to ask a few questions if he or she has any. That eliminates disputes over the reliability of the system or the process.

Mary Bennett

Thank you, Judge Kim. Caroline, I’m going to shift this to you. When we’re looking at the evolution of technology over time, and manual review to TAR and now GenAI, and how that’s shifted the skillsets of your team, how do you see those skills changing? What kind of skillset are you encouraging your team to learn now with all these AI tools? And how have you had to lead the charge of them to acquire those new skills, especially as we’re all learning in real time together as well?

Caroline Sweeney

Yeah. I mean, it requires some skill enhancement. So, they become not just proficient but experts in how to apply the various analytic tools available to us in our platform to consult with legal teams and clients on how to use those tools to drive down a population for review and manage costs. Now, we’re very centered around getting proficient with generative AI, making sure that they understand how to prompt to get the right kind of responses, and how to vet the responses that they’re getting from the generative AI tools. It’s developing new workflows, processes, and procedures. We saw with TAR that people had to learn not just about how to leverage the various analytic tools and do ECA, early data assessment, but also they had to learn something a little bit about statistics. The one math course I ever took that I was glad I took was predictive coding and continuous learning. It’s really important to educate yourself on what’s out there, what the different tools are, what can be done, how we can leverage them, and how to leverage training opportunities. For example, we’ve brought in a firm-wide GenAI training platform so that attorneys can get CLE credit while learning about GenAI technology and how to do prompting. For example, that’s a requirement if you start using generative AI in our transcript platform. So, all of that is an ongoing process to develop those skills. And personally, it’s what I love about this industry because you’re constantly learning new skillsets and having to apply those, and again, showing or demonstrating that you add value for the legal team to be a resource to them.

Mary Bennett

Thank you, Caroline. Mary, the EDRM, as we all know, works with organizations, law firms, and corporations. You also work a lot with people entering the field or looking to pivot. So, right now, where we are, how do you advise folks on acquiring new skills or doing different training? Where do you see individuals needing to allocate their time or efforts to upskill in this given time?

Mary Mack

Well, it involves a bit of self-assessment and, as Caroline said, a little bit of math these days. With predictive coding or with GenAI systems, validating what comes out and being able to communicate about it in a meaningful way, most attorneys, and probably paralegals, didn’t necessarily major in math. Getting a grounding in that will provide a foundation for where we’re moving; I would say, in the far near term, which is into quantum computing, which is all about probabilities. It will no longer be binary off/on; it will be more probabilistic, and that will be expressed in percentages. Oh my gosh. That’s where I suggest that people get a good grounding, and then the other aspects of technology will come more naturally.

Mary Bennett

I feel bad for myself and my fellow English majors out there, where math isn’t our forte, but important nonetheless. Adam, I want to go to you because we’ve talked a lot about GenAI, what it truly means right now, and why it’s pivotal for our industry. Do you think our industry is at a true inflection point with this technology right now? Are we still on the hype cycle? What’s your take on the potential of GenAI at this moment?

Adam Rubinger

I’ll take us back slightly. Looking back, one of the things that we saw was Judge Kim talking a little bit about adoption and explaining technology. It took us a while to accept TAR 1.0, 2.0, whatever. They both took some time. And why I think, to answer your question, we’re at an inflection point and not really at a hype cycle is that the acceptance gap between when a technology first appeared and when it’s being used, and I would say even somewhat mainstream now, has been a year or less. We’re seeing clients now asking about it. Almost every meeting I have these days with our clients inevitably comes up. They’re interested. They want to understand what it does, how it works, what it costs, how it can be valuable, and what efficiencies can be gained. We’re starting to work on projects now with clients, and we’ve launched several AI tools, solutions, and technologies that will take us to a whole level of efficiency that we haven’t seen yet. We saw many efficiencies as TAR started to evolve and refine. And it hasn’t changed a lot in the last 10 years, right? From my perspective, this technology, and Mary talked about it, moving into advanced models and advanced computing capabilities will take us to a whole new level. I’ve had comments from clients saying, “Why would we ever not use this?” once they’ve had the opportunity to use it. It will be interesting to see the evolution over the next couple of years because, if you think about it, it’s only been about 18 months since ChatGPT went mainstream. People started using it and talking about it. And everyone’s interested. Those who are more technology-forward are beginning to use it, and the ones who are using it are seeing incredible results.

Mary Bennett

It’s hard to imagine what life was like 18 months ago before it. That certainly expedited how I work. Dennis, what about you at DLA? I know you’re part of that group. What are your thoughts? Is this a true inflection point? What are you seeing from your lens?

Dennis Kiker

I have to agree with Doug; it’s absolutely an inflection point. I think that AI in our firm is integrating a lot of, I mentioned, a lot of different practice areas from transactional practice, doing contract analysis. We have a team that’s building small language models and large language models. It’s interesting. For us, this is an EDRM, so we’re interested in eDiscovery. And I do think that this is a true inflection point, and we’re going to see acceleration, as Doug mentioned. If you look at our progress, we started in a business where everyone was watching every document. They used to be paper. They transitioned to electronic, but somebody looked at every document and made a final coding decision. We then transitioned to TAR, where we said, “Okay, we’re going to look at every likely responsive document, but let the technology help us decide what not to look at and what to exclude.” With TAR or AI, we are at a point where we may get to the step where we don’t need to look at every document. We can allow an algorithm to decide what is responsive and what is not responsive, and we can rely on effective quality control to catch privilege and do those sorts of things. I don’t think that’s a long way off. It shouldn’t be the 10 to 15-year gap we had for TAR to mature. It’s because of what we talked about earlier and some of what Doug said. This is a much more generally accepted technology. It’s something that is part of all of our lives every day. The key is that people are already comfortable with AI in other aspects. TAR is not something people conceptualize. Certainly, it existed on Netflix and in other areas, as somebody mentioned, but it’s not related to document review. People don’t think about that. TAR’s AI is different. Everybody is exposed to AI. Everybody’s using AI for several reasons. The transition from where we are today to where we will be will be much faster. We’re a legal industry, so it’s probably going to be years. It’s not going to be 15 years, I hope. I’m an eternal optimist. The glass is always all the way full. So, I could be proven wrong. I thought TAR would accelerate faster than it did. But yeah, we are positive about an inflection point here, and it’s exciting. Somebody said it, Caroline said; this is an exciting industry to be in, and it’s never been more exciting because of technology’s potential.

Mary Bennett

Thank you. Shifting gears slightly but still in the same vein of so much going on in our industry, there’s just so much data out there. We discussed it today using collaboration tools, mobile phones, and email. Doug, I’m sure you write about this a ton in your day-to-day. When we look at all the data scattered across these platforms, how do you think organizations should think about discovery readiness in a way that scales with the reality of just how much data is out there?

Doug Austin

They say the three Vs of big data are volume, velocity, and variety. The last two have become as important, if not more important, than the volume. This emphasizes the importance of information governance in our industry and understanding your data, the types of data you have, the locations where it’s located, with data maps, and things like that. It also suggests using technology to help leverage those abilities and understand the use cases for it. You’ve got to develop all that understanding before you have a case. You’re already behind if you start discovery readiness when the case is filed. And I think if you look at the importance of InfoGov, you just have to look at the EDRM model. When the EDRM model first came out, there was no box, circle, or anything for InfoGov. Then, they eventually added a small box called information management, which turned into a small circle called information governance. Finally, they have the full IGRM model on the left side, which is one of the few instances where there’s a model within a model. But, almost really, you could transpose them because that’s how important IGs become. It’s become the foundation of eDiscovery, cyber, data privacy, risk management, and, of course, AI. Bad data means bad AI. That’s the key. You always want to continue in an InfoGov program to look at the data out there. New data types are always being defined. Now, all this generative AI-generated data is becoming discoverable in litigation. Having a strong InfoGov program and a process where you’re keeping it evergreen is key to keeping up with all these continually evolving data types.

Mary Bennett

We could discuss information governance; there’s just so much to discuss, but it is definitely an important part.

Doug Austin

Absolutely.

Mary Bennett

I’ll open it up to the other panel because this topic is important. Does anyone else want to expand on or react to Doug’s comments about the importance of information governance?

Caroline Sweeney

I completely agree with that. And I feel like InfoGov had a moment a while back and then sort of faded. But that moment has returned, and it’s really in full blossom, I guess I’d say, because the need to really have a handle on what you have and where it’s located is as important as ever, and it’s ever challenging. Once you know what you have and where it is, how will you collect it? How are you going to handle versioning? How are you going to handle linked attachments? Can you collect it? We do regular monthly eDiscovery boot camp sessions for our attorneys and paralegals. We just spent an hour program last month with someone in our Government Investigations group, a partner from that group, an outside consultant, and my team talking about the things that the attorneys have to be able to discuss when facing these issues with clients, etc. Then, of course, it flows down to how you review it. How do you unitize the chat so that you can review those effectively? How can you easily produce? And even last night, I was at 9:00 last night emailing two different legal teams, talking about how we authenticate video evidence and poke holes in this video evidence that the other side’s trying to introduce. Then, we were asked if we had Owl technology to display a video in court during the trial. It permeates the whole EDRM; again, you have to be on top of it and educate yourself all along the way. 

Adam Rubinger

One of the things that we’re seeing, if we’re talking about technology, is really it’s been a challenge forever. Back in the day, and Mary will remember this: when we came out with our litigation readiness assessment, we would go off to corporations and ask them a bunch of questions about what they know about their data, where it’s stored. Even today, we’re still getting answers we do not know about. And there’s a lot of rot out there. There’s a lot of data floating in many different areas, and people don’t have a good sense of it. The good news is that we’re working on utilizing AI to do things like classifying data. The cloud has allowed companies to centralize where their data sits, whether in Google, Amazon, Microsoft, or whatever cloud-based environment. These tools are being developed to help classify that data. As data is being created, it’s being classified. You’re also going to see information governance really at an inflection point, where we’re going to be able to find documents very quickly. They’ll be auto-classified based on machine learning and AI learning. That will be a boon for ensuring that the data you don’t want is classified and trashed. The data that you want to keep is utilized in a way that you can make better business decisions, you can make better legal decisions. There are many ways these technologies will be handled in place so that it’s not a human going in and saying, “This is what this document is. This is what this document is.” Suppose you prompt ChatGPT now and ask for information about it, especially in an eDiscovery case, litigation, or investigation. In that case, you can ask it about classifications, and it will classify it. It’ll tell you, “These are board meeting notes. These are recordings. These are contracts.” These are all the different types of information that these documents have based on the content, not anything other than the large language models and the machine learning models. I have seen so many of these documents over and over and over. It begins to understand what this is. How is this important? Is it not important? Is it sensitive? All this is going to be done and is being done, and it is indeed being worked on in data lakes where we store our data, which is currently in place. So, as eDiscovery professionals, we will see a day in the future when data will come to us for litigation that is already classified, already deemed potentially responsive, and already deemed potentially privileged because it knows what privileged communications are and who the actors are. We’re sitting here reactively thinking about eDiscovery and the data that comes to us for all these things. It’s moving upstream, where we’ll see some exciting changes.

Caroline Sweeney

Not only that, but it will also arrive at you already redacted, where it identifies the PII content and whatnot.

Adam Rubinger

Yes, yes, exactly.

Dennis Kiker

I can chime in on that. I agree with the promise. I agree with the potential. The one I see as a practical matter is that the people like us who need that information to contribute to those decisions about what’s PII and potentially privileged often don’t happen. One of the things companies need to think about more than ever it’s always been the case. Still, more than ever, the legal team needs to be involved in introducing new technologies so that these kinds of ideas, these kinds of, “We know what’s going to happen down the road sooner or later. I have bad dreams about what we will do, and we have to produce our props for all of the AI systems we’ve had. Are they preserved? Are they even preservable?” All of those sorts of things. If we don’t bring our legal teams into the conversation early on, the applications and technologies will be awesome for business purposes. But when it comes time to identify, preserve, and collect that information, we’re still trying to figure out what to do with M365, and that’s not new technology. I worry that most companies, or at least many companies, don’t have that ongoing conversation with the legal team to make sure that we are prepared to do the job that we will have to do at some point in time.

Mary Bennett

Thank you, Dennis.

Doug Austin

Part of the reason we are trying to figure out M365 is that it’s constantly changing. It’s like a moving target when it comes to the data in M365 and the capabilities and what have you. So, that’s part of the challenge, too.

Judge Kim, I want to go to you and look at AI and what’s produced in court. In an earlier conversation, we were talking about this call. You mentioned that AI-generated priv. logs might actually be more objective than human ones. So, as a judge, what do you need to see, either from a procedural standpoint or evidentially, for these outputs to be viewed as reliable in court?

Judge Young Kim

First, I don’t think there’s anything to prepare evidentiary-wise because a privilege log would not be used as evidence. However, a privilege log will become a thorny issue in the event of opposing sides, as it’s overinclusive. In terms of the procedure, this is where generative AI can be very helpful in saving time and money for clients, where the algorithm can perform tedious tasks to identify privileged documents and provide all the details that are necessary under Rule 26. That tedious task can be eliminated. Since I have been on the bench since 2010, I’ve never seen a privilege log that complies with Rule 26. Now, I’m not looking at all the privilege logs; I’m only looking at those problematic privilege logs. Taking away that human judgment might provide the other side some level of comfort that there wasn’t a biased judgment in play. For that reason, generative AI can be more trustworthy in terms of privileged logs. But something that Adam said that I think I need to say something about in terms of documents being coded in a certain way so that they can be collected more easily or they are withheld more easily, I think that the companies, when they start doing that, there has to be a very clear record of what terms or what parameters you are using to code these documents. Because I can see from a plaintiff’s perspective, and if, in fact, a company basically says, “Just go ahead and code everything privileged as long as there’s a word confidential in there,” then as a plaintiff’s lawyer, I’m going to be very concerned with that. Judges would be more receptive to AI and algorithm-driven protections and productions by then. And plaintiffs should be able to say, “Wait a second, that’s overinclusive based on the parameters you used.” To avoid the issue of not having the information available, I think companies need a standard protocol to ensure that all parameters are preserved. I don’t know if I got through all of this. My connection isn’t all that great in this hotel room.

Mary Bennett

That sounds good to me. Does any panelist want to respond to anything Judge Kim said before we move on? Judge Kim, staying with you here, I know Dennis was talking about how we might get to a point where we don’t need to review every single document with TAR or what have you. From the bench’s point of view, do you think there might be a case where manual review could be viewed as unreasonable or insufficient in some manner? Do you think we’re at that point?

Judge Young Kim

The short answer is yes. I have not seen an argument from the other side that the production is incomplete or unreliable because it was done manually. However, I have had situations where I will push back on the defendant seeking more time for manual review. And, of course, I raise my eyebrow and say, “Wait a second, you’re still doing manual review. What about doing TAR to speed things up?” These days, plaintiffs are more receptive to TAR to get the production quickly. And so, when somebody says the manual review doesn’t coincide with the discovery schedule, I will push back. But I’ve not had a motion filed requiring technical-assisted review. However, if the issue arises, an argument can be made that manual review is inferior because empirical studies have shown that manual review is indeed inferior.

Mary Bennett

That’s interesting. Thank you, Judge Kim. I’m going to start with you, Dennis. We’ve been talking about the evolution of the industry, and all these new tools like GenAI coming into the mix, and many opportunities come with integrating these new tools. However, there are also some risks that we haven’t talked about ad nauseam here, but we all know that there are risks with some of these news tools. When you’re looking at yourself or yourself implementing tools like GenAI, how can you innovate responsibly while ensuring that you’re maintaining defensibility and trust?

Dennis Kiker

It’s a really good question. As technology becomes more complex and more difficult to understand, the people who are creating large language models—the technology we’re talking about relying on in the future—don’t understand how it works either. So, that creates a real problem for lawyers who have to go into Judge Kim’s court or explain to a client what they’re doing. It comes back to something folks brought up a little bit earlier. We focus too much sometimes on how it works. How does a support vector machine work? I don’t know. I know the words and generally know that three- or four-dimensional space in which it works, but I don’t know what it’s doing. I can conceptualize it. It becomes more difficult with large language models. But what works are statistics and metrics that we can use to evaluate the outcomes? We can test them, and we can do our quality control, making sure that the outcomes are what we expected. We’re going to become increasingly dependent on and reliant on statistical analysis to defend the results of a process, as opposed to diving into the details of a process. And I think this is a good thing. I’m often called upon to negotiate with the plaintiff’s counsel. I’m a defense lawyer, so it’s the plaintiff’s counsel. Let’s talk to opposing counsel about how our process is going to work. What are you going to do? How are you going to set up your TAR process? How were you going to pick your seed sets in the old days? All of this is getting into the weeds of how the process works, when the real proof in the pudding is what the outcome is. Let’s sample the outcome, test it using statistics that have worked for generations, and figure out whether or not the outcome is right, whether it needs to be tweaked, and whether the prompts need to be adjusted. I think that’s where we’re headed. We can innovate and embrace these technologies if we don’t abandon the idea that we still have to test the outcome. Anybody who’s gone through the process of prompting an AI application knows that prompting is an art. It’s far from being a science. I learned this. We used one of our in-house applications in a parallel test to see what would happen if the AI-selected documents were responsive to certain interrogatory requests compared to humans. I created the prompt and thought I did a pretty good job. The results looked pretty good. Then the prompt engineer came in and tweaked my prompts with this particular application, praising the application and giving it positive feedback in your prompt to help the results. And they became much richer, much better, much more detailed. And so, that is an art, and the only way to test that art to see if it’s working is to analyze the results, and you do that through statistics.

Mary Bennett

Art and science. So, I like that, Dennis. We have a few minutes left, and I would love to just get to the last question here. And I’ll start with you, Adam. If you could give one piece of advice, what would you like the audience to walk away with to really help them lead our industry with things changing so quickly?

Adam Rubinger

I would say to spend a lot of time using technology. There are a lot of training and certification opportunities out there. The technology itself is going to be the driver. Technology certainly supports change workflows. People can use an incredible number of resources to stay abreast. Dennis said it perfectly: this stuff is an art. It isn’t science. Prompting is an art. Depending on how you prompt it, if anybody’s spent time playing with ChatGPT, it’s garbage and garbage out. And so, practice using these types of technologies because they will become pervasive, not just in eDiscovery but in your lives, on your phones, on your computers, and in your work. It’s going to be everywhere because it’s going to make us better, make us faster, make us more efficient, and certainly hopefully do our jobs better. I completely reject the notion that AI is going to put us out of work. It’s going to enhance our jobs, create new jobs, and make the work that we all do interesting. We’re going to see some amazing transitions over the years.

Mary Bennett

Thank you. Mary Mack?

Mary Mack

Well, just like when we moved from Boolean and needed to train people to create very elaborate Boolean queries, people who would translate those into SQL code lost their jobs or lost that part of their job when we moved to a better interface. We’re in a situation like that. Stay aware of what can be automated with these new tools. Help your colleagues and stay aware of the new opportunities emerging so that people can move quickly in the revolving door of eDiscovery, from providers to law firms to corporations to government. We all move around, and this is going to be a time of increased mobility. And so, stay curious and stay compassionate. That’s what I would say.

Mary Bennett

Love that. Doug, what about you, my friend? 

Doug Austin

I was going to say be curious or stay curious. Curiosity may have killed the cat, but it’s a terrific trait for an eDiscovery professional to have. Set aside five to 15 minutes daily to read about the topics or listen to a podcast or what have you. Attend one free webinar a month. There are plenty of them out there. EDRM has a great program. HaystackID has a great program. Continue to educate yourself each day and each month. And before you know it, you’ll know a lot. And then find time to play with the tools because they’re exciting, and you learn a lot by doing that.

Mary Bennett

Thank you. Caroline.

Caroline Sweeney

Get comfortable with the AI and develop prompting skills. In our case, we have an entire business model built around document review and eDiscovery services. So, other law firms or vendors do. It’s really important to bring people along as part of the journey, not just the people providing those services, but also bringing our lawyers along so it’s not scary technology that they can see the value and the benefit. The constant education: I love what Doug said about attending webinars. We do a lot of vendor presentations so that we understand what’s out there in the market. So, you do not have to make it scary. Make it fun. Be on this journey together. It’s not going to get rid of our jobs. It will change what we do, and it’s an opportunity to reinvent ourselves in some ways.

Judge Young Kim

I’ll highlight what Adam said about becoming more familiar with whatever platform or technology you use. But when doing so, you may want to have a red team or at least that mindset. So, if I use this particular platform for this objective, what will the opposing side question? How are they going to react? And how are they going to perceive the byproduct? From a bench’s perspective, I’m interested in disputes being resolved or not having disputes. Having that mindset as you get familiar with the platform or technology is very helpful in dealing with the opposing side. I don’t think it’s just good enough to say, “This is good for my objective, and therefore, I’m going to use it,” and that’s the end of the discussion. You still need the other side’s buy-in to ensure that you’re not actually causing more resources to be spent because of your use of that technology. 

Mary Bennett

Thank you. And last but not least, Dennis.

Dennis Kiker

I’m going to echo what folks said. Just enjoy this business and this industry that we’re in. I’m an eDiscovery lawyer because fate didn’t let me become a trial lawyer, so I pivoted to eDiscovery because I wasn’t in court all day. And I have loved this industry for the last 25 years. There are so many interesting things going on. So, enjoy it, embrace it, and don’t let your day-to-day busyness get in the way of being curious, as folks say. That was a great, great piece of advice.

Mary Bennett

Thank you. I want to thank all of our speakers here today. I can’t believe it’s already been 60 minutes. I really, really appreciate your time and expertise. And to our audience, thank you for staying with us. You will receive a copy of this presentation. We, HaystackID and EDRM, thank you both for coming. EDRM, thank you so much for hosting, and we will see you all next time.

---

Expert Panelists

+ Doug Austin
Editor, eDiscovery Today

Doug is an established eDiscovery thought leader with over 30 years of experience providing eDiscovery best practices, legal technology consulting, and technical project management services to numerous commercial and government clients. Doug has published a daily blog since 2010 and has written numerous articles and white papers. He has received the JD Supra Readers’ Choice Award as the Top eDiscovery Author for 2017 and 2018 and a JD Supra Readers Choice Award as a Top Cybersecurity Author for 2019. Doug has presented at numerous events and conferences, including Legaltech New York, ILTACON, Relativity Fest, University of Florida E-Discovery Conference, Masters Conference, and many local and regional conferences. Doug has also presented numerous CLE-accredited webcasts.

---

+ Dennis Kiker
Senior Attorney, DLA Piper

Dennis Kiker is a litigation attorney with extensive experience in eDiscovery. His focus is on guiding clients and litigation teams on strategy and implementation of defensible practices associated with the identification, preservation, analysis, review, and production of documents and data, designed to help our clients realize better results in less time and at a lower cost. Dennis is among the firm leaders in piloting and implementing AI and other advanced technologies to improve outcomes and lower costs. Dennis has a broad range of industry experience, including pharmaceutical and medical device, automotive, financial services, insurance, industrial manufacturing, and technology; the experience of a seasoned litigator; and a lifelong interest in advanced technology. Dennis also brings a unique perspective to the eDiscovery practice. He has worked with companies to develop complete eDiscovery response programs, providing him with an intimate understanding of the in-house challenges associated with the management, identification, preservation, and collection of information. He has also managed teams of technical professionals in handling the generally outsourced tasks of processing, hosting, and analysis, giving him an in-depth understanding of how eDiscovery professionals can best support litigation teams. Lastly, Dennis has represented clients in state and federal courts around the country on discovery matters.

---

+ Hon. Judge Young Kim
Magistrate Judge, Northern District of Illinois

Judge Young B. Kim is a U.S. Magistrate Judge for the Northern District of Illinois. He was born in South Korea and his family emigrated to the United States when he was 11 years old. He is a graduate of the University of Illinois at Urbana-Champaign and Loyola Law in Chicago. He began his legal career as an Assistant Cook County Public Defender in 1991. From 1993 to 1995, he clerked for District Judge Charles R. Norgle of the Northern District of Illinois. From 1995 to 2001, Judge Kim served as an Assistant U.S. Attorney with the Chicago office, prosecuting and litigating both civil and criminal cases. In 2001, the Equal Employment Opportunity Commission appointed him to serve as an Administrative Judge. Then in 2010, the District Court for the Northern District of Illinois appointed Judge Kim to a magistrate judge position. As a federal judge he particularly enjoys presiding over citizenship naturalization ceremonies in the ceremonial courtroom where he himself was naturalized as a United States citizen in 1986.

---

+ Mary Mack
CEO, Chief Legal Technologist, EDRM

Mary Mack leads the EDRM, a project-based organization, and is the former Executive Director of a certification organization. Mack is known for her skills in relationship and community building as well as for the depth of her eDiscovery knowledge. Frequently sought out by media for comment on industry issues, and by conference organizers to participate, moderate a panel, lead a workshop or deliver a keynote. Mack is the author of A Process of Illumination: The Practical Guide to Electronic Discovery, considered by many to be the first popular book on eDiscovery. She is the co-editor of the Thomson Reuters West Treatise: eDiscovery for Corporate Counsel. Mack was also recently honored to be included in the book, 100 Fascinating Females Fighting Cyber Crime, published by Cyber Ventures in May 2019. Mack has been certified in data forensics and telephony. Mack’s security certifications include the CISSP (Certified Information Systems Security Professional) and the CIAM (Certified Identity and Access Manager).

+ Adam Rubinger
Chief Client Experience Officer, HaystackID

Adam Rubinger is currently the Chief Client Experience Officer at HaystackID. In this role, Rubinger works directly with HaystackID’s corporate clients on their eDiscovery team as the executive sponsor. He provides guidance, consultation, and advice to the teams in all areas of the process, from collections to production. He is highly involved in most aspects of the portfolio and works at all levels of the organization. Given both his legal and technical background, he is an excellent liaison between the HaystackID team, the client, and their outside counsel.

+ Caroline Sweeney
Director, Knowledge Management and Innovation, Dorsey & Whitney LLP

Caroline Sweeney is responsible for the delivery of Dorsey’s e-discovery services, including LegalMine Managed Review services, litigation technology support, and trial technology support. Caroline is a member of Dorsey’s Electronic Discovery Practice Group and the Cybersecurity, Privacy, and Social Media Practice Group. She has extensive experience consulting with attorneys and clients with regard to e-discovery, including identification, preservation, collection, processing, review, and production of electronically stored information. Her 25+ years of experience in the litigation support industry include working in the law firm and litigation support vendor environments. Caroline is actively involved in the e-discovery community. She is ACEDs certified, currently Co-President of the Twin Cities ACEDs chapter, and participated in the development of the litigation support certification test for the Organization of Legal Professionals (OLP). In addition, Caroline is a current member of The Sedona Conference Working Group on Electronic Document Retention and Production and sits on the Information Governance steering committee for the International Legal Technology Association (ILTA), and recently co-chaired the Minnesota E-Discovery Project Working Group 5 on Use of Technology to Facilitate Production of E-Discovery. She has served on a number of e-discovery vendors’ client advisory boards and has experience with a wide array of e-discovery processing and review platforms. In addition to various software certifications, Caroline has received additional certifications in the areas of Legal Project Management and records and information management, and is also listed in Strathmore’s Who’s Who. She has participated as a presenter and faculty member at various conferences on litigation support and E-discovery and is a former instructor of “Computers in the Law” at the Minnesota Legal Assistant Institute. In the early 1990s, she was a co-founder of the Minnesota Association of Litigation Support Managers.

+ Mary Bennett [Moderator]
Senior Director, Content Marketing, HaystackID; Senior Director, Content and Community Initiatives, EDRM

Mary Bennett, HaystackID’s Director of Content Marketing, focuses on the power of storytelling to educate the legal technology industry on pressing issues impacting practitioners. With nearly 10 years of content marketing experience, Bennett joined HaystackID after working at an agency to help B2B tech startups grow their marketing engines through content that drove audiences through the marketing funnel. Before her agency experience, Bennett worked at Chicago-based Relativity as a Senior Producer on the Brand Programs team. She was a founding member, host, and producer of Relativity’s Stellar Women program and producer of the company’s documentary series, On the Merits. In her role, Bennett crafted and socialized important stories that elevated the eDiscovery community and illustrated technology’s potential to make a substantial impact.