[Webcast Transcript] Protect Sensitive Data and Control Costs: An eDiscovery Blueprint for the Construction Industry
Editor’s Note: The recent HaystackID® webcast, “Protect Sensitive Data and Control Costs: An eDiscovery Blueprint for the Construction Industry,” explored the challenges of managing vast and complex construction data while ensuring compliance and cost control. Experts shared insights on leveraging AI-driven solutions, including generative AI (GenAI), domain analysis, and technology-assisted review (TAR), to streamline document review and reduce inefficiencies. The panelists shared how AI enhances early case assessment, improves keyword identification, and strengthens privilege review processes, ultimately driving more intelligent eDiscovery workflows. As data volumes grow and regulatory scrutiny increases, legal teams should adopt AI-driven strategies to mitigate litigation risks and manage sensitive project data effectively.
Expert Panelist
+ Esther Birnbaum
EVP, Legal Data Intelligence, HaystackID
+ Mark MacDonald, CEDS (Moderator)
VP of Enterprise and Strategic Accounts, HaystackID
+ Sam Morgan (Moderator)
Director of Legal Solutions, HaystackID
+ Young Yu
VP of Advanced Analytics and Strategic Solutions, HaystackID
[Webcast Transcript] Protect Sensitive Data and Control Costs: An eDiscovery Blueprint for the Construction Industry
By HaystackID Staff
Managing data in the construction industry presents unique challenges due to the sheer volume, complexity, and sensitivity of information generated throughout a project’s lifecycle. From early planning stages to project execution and potential litigation, legal teams must navigate diverse data sources, compliance requirements, and evolving cybersecurity risks. In the recent HaystackID® webcast, “Protect Sensitive Data and Control Costs: An eDiscovery Blueprint for the Construction Industry,” industry experts explored how AI-driven eDiscovery strategies can help streamline data management, reduce costs, and enhance defensibility in legal proceedings.
During the discussion, HaystackID’s Esther Birnbaum and Young Yu shared insights on leveraging advanced analytics, technology-assisted review (TAR), and generative AI (GenAI) to optimize document review workflows. By implementing strategic tools like domain analysis and AI-powered privilege review, legal teams can significantly reduce irrelevant data, improve accuracy, and increase efficiency when responding to subpoenas, government inquiries, and construction disputes. The experts emphasized how AI enhances early case assessment and enables more precise keyword identification and deeper insights into data classification, helping legal professionals make informed, defensible decisions.
Throughout the expert presentation, HaystackID panelists shared how construction firms and their legal teams can proactively prepare for litigation by incorporating AI-assisted active learning and structured workflows tailored to their industry. Read the below webcast transcript and watch the recording to get actionable strategies for managing complex construction data while maintaining compliance and cost control.
Transcript
Moderator
Hello everyone, and welcome to today’s webinar. We have a great session lined up for you today. Before we get started, there are just a few general housekeeping points to cover. First and foremost, please use the online question tool to post any questions you have, and we will share them with our speakers. Second, if you experience any technical difficulties today, please use the same question tool, and a member of our admin team will be on hand to support you. And finally, just to note, this session is being recorded, and we’ll be sharing a copy of the recording with you via email in the coming days. So, without further ado, I’d like to hand it over to our speakers to get us started.
Mark MacDonald
Thank you, Mouna. Hi, everyone, and welcome to another HaystackID® webcast. I’m Mark MacDonald, one of your expert moderators for today’s presentation and discussion, “Protect Sensitive Data and Control Costs: An eDiscovery Blueprint for the Construction Industry.” I’ll be co-moderating with my friend and colleague, Sam Morgan. Hi Sam.
Sam Morgan
Hi Mark.
Mark MacDonald
This webcast is part of HaystackID’s educational series, designed to help you stay ahead of the curve regarding cybersecurity information and eDiscovery objectives, especially in the construction field.
As Mouna mentioned, we’re recording today’s session for on-demand viewing, and we’ll make the recording and transcript available afterward. Sam and I are looking forward to presenting today alongside our colleagues, Esther Birnbaum and Young Yu. And we’re going to have a comprehensive discussion. This is the first in our eDiscovery series on construction, so welcome to it; we’re happy you’re here. For those of you who are here live with us, thank you for joining. For those of you who might listen to this at 2X sometime in the future, thank you as well. That’s my protocol when it comes to webinars. So, let’s do some quick intros. Sam.
Sam Morgan
Sure. Mark, as he said, is actually my colleague with over 20 years of eDiscovery experience. He’s ACEDS certified and an expert in construction litigation and eDiscovery. Mouna, do you want to go to the next slide?
Mark MacDonald
It is my privilege to introduce my colleague, Sam. Sam and I together lead the construction efforts here at HaystackID. Like Sam said, I’ve been doing this for a long time. Sam has tremendous experience working in this industry across multiple sectors and different verticals. But Sam and I were recently at the construction conference out in Las Vegas. Shout out to that event, which was fantastic. And before I move on and we do intros for Esther and Young, let me pinpoint that our industry, I call it our industry, the construction industry, is just so wonderful, between the people that we work for, which are all the attorneys inside the corporations, the outside counsels, the community of construction is just phenomenal. I just want to point that out and say how happy we are to be here and to help contribute to what really is an important issue when it comes to construction. What do you do when you have a legal event? They’re complicated. And here we are in the middle of intros, I’ll stop right there before I get too carried away. But let’s introduce Young and Esther quickly.
Sam Morgan
Yeah, sure. Mouna, next slide. With privilege, I introduce Esther Birnbaum. Esther is an industry expert who has vetted, analyzed, worked with, and created models around both complex litigation and developed expertise in AI, which is on the tip of everyone’s tongue these days. Esther is really our internal and external resource for helping every one of our clients who has questions about the budding industry of AI from both a practical and a theoretical standpoint. I am often amazed at how she knows almost every AI platform. And I’m thrilled that she’s on our panel with us today.
Mark MacDonald
And Young is our VP of Advanced Analytics and Strategic Solutions. Young and his team inside HaystackID are the go-to consultants that are really, not to use sales terms, at the leading edge of AI and how we’re using it in the eDiscovery and legal industries to support our clients. I know in construction that AI is being used all across projects from design-build to BIM models, and it’s certainly spilling over into eDiscovery and how we can help attorneys sift through mountains of data quickly, efficiently, and defensively. Even though this is an inaugural webinar for construction where we want to do an intro and touch on points across the whole spectrum of eDiscovery, as you can see by our panelists here, we spend quite a bit of time on AI. Because time and time again, when we talk to our clients, “Hey, what do you want to hear?” We keep coming back to, “Save us money, reduce data sizes, help our outside counsels be efficient.” Reducing efficiency is really, in my personal opinion, inefficiency is the biggest cost component of eDiscovery; it probably applies to the build of projects as well. That’s where we will focus more than half our time today. So Young, Esther, I’m so happy to have you here.
Sam Morgan
Mark. Absolutely. And just before we move on, I want to point out something else about Young and his team. AI is the one thing everyone’s talking about today, but I can tell you through dozens upon dozens of engagements that Young’s team has also leveraged all types of analytics. And they are experts in that space. It goes exactly where Mark was talking about reducing data size and overall costs. We also lean heavily on Young and his team for that. I just wanted to point that out before we move on.
Mark MacDonald
Yeah. Okay, Mouna, we can move on.
Sam Morgan
So today, obviously, we will talk about construction litigation specifically but complex eDiscovery globally. With that, for everyone who logged into this webinar today, data size and volume are some of the challenges with construction eDiscovery or any complex eDiscovery. So we’re going to touch on a bit of that. We will talk about the Cybersecurity Maturity Model Certification (CMMC) [as it relates to] compliance and cybersecurity. Most importantly, and from Mark’s point of view, Esther’s point of view, and Young’s point of view, we cannot be successful without a well-vetted, thought-out blueprint that’ll be our planning document going forward. We’ll also talk about what has worked to this point in construction eDiscovery, and we’ll talk about what’s coming on in the future as well. Mouna, you can go to the next slide.
Mark MacDonald
All right. The data problem. We should say the data problems or the challenges in managing construction data. So we’ve got four points here: volume, complexity, sensitivity, and litigation. Start with volume. We all know that in construction, we have enormous data sizes. It goes back to the owner of the project who conceptualizes and begins things years before they ever go to design, before engineering gets involved, before the ground gets broken, way before we ever consider litigation typically or a legal event that might be tied to one of these projects, whether it’s a subpoena, a government inquiry, it could be an injury on the work site. So many things can be implicated when it comes to data and eDiscovery and construction. Our role as a trusted vendor to our clients, and whether that’s a construction, a builder, usually a company, a corporate company, or their outside counsels, we want to help with our knowledge and what we know. How do we do things like use data analytics, tricks of the trade like, hey, we’ve got 250,000 records that went through culling and deduplication and keyword searching and date range filtering, and now we’re facing down the barrel of a massive doc review? TAR has been a great tool for us over the past 10 years. It’s been embraced by law firms, corporations, court systems, and experts. And now, as we keep talking about, we’re entering a new world of AI. Back to tricks of the trade, doing things like running a domain analysis is one of our favorite things. So, not only have we done all of the normal things that happened during the prepping for your discovery and prepping for the doc review, but Esther, you, and I were on the phone with a client just on Friday last week discussing this very thing with that pretty much same similar document review that this particular lawyer’s looking at. Running a domain analysis is a simple thing: ask your eDiscovery vendor, whoever’s running your data and processing it, to give you a list of all the different domains. Now we’re talking NetJets, all the fantasy football, and Amazon because we’re human beings and like to cross-pollinate our work environment with our home and personal lives. By doing something like this and agreeing with outside counsel, look, we can get rid of sometimes 20,000/30,000 completely irrelevant emails that might’ve hit on a search term and might’ve hit in the date range but are certainly not relevant to this case. And you can watch the pounds fall off; you’re shedding weight like you’re using a weight loss drug in some cases. And so it’s really quick and effective. Sorry, Sam, I know I wasn’t supposed to make that reference, but here we are. And so make a note of that. So when you’re looking down the barrel of the next large litigation, a simple thing like running a domain analysis. Sam, if there is anything to add there, please feel free.
Sam Morgan
Absolutely. And with the data problem, what we try to do for our clients, with the help of our internal teams, folks like Esther and Young, is we try to bring quiet to the noise. We’re dealing with complexity from various data sources. We’re also leveraging that with the sensitivity of proprietary project management software or potentially privileged documents. And also, look at this sometimes in a silo of a specific defect or across an entire project that may touch upon multiple subs and primes as well. But really, we try to bring calm to a situation filled with angst; that is our expertise.
Mark MacDonald
And high dollars, too. You’ve kind of jumped a little bit into the complexity aspect of it as well. But we’re always dealing with numerous parties and subcontractors. Where does the data live? Do the project managers use their own project management software? What information do we need from that? How do we get to it? It’s not just in our world taking data, processing it, and putting it into a Relativity or other website for the attorneys to go bang away at. Again, back to the plan, we want to understand, is this a subpoena? If so, are we a third party to it? How do we get the most relevant information for the requesting party? Be compliant with the subpoena without spending a fortune. Is it going to be a keyword search and date range filter? Look for other project names in there to ensure we’re not giving away any information that might lead to other adverse actions for our clients. Certainly privileged, but we also don’t want to spend a lot of money doing a proper proof review. Esther, Young, this is again more fodder for your discussion downstream. But these are the things we think about when we get that phone call from our clients. CAD drawings, BIM models, and so on and so on. Will it be a financial investigation into how this was funded or any inquiry? In construction, it can be anything and everything. So Sam, sensitivity, I’ll turn that over to you.
Sam Morgan
Yes, well, actually, Mark, I think we touched on sensitivity already when we talked about various forms of proprietary software and privileged documents. And then we also try to make sure our corporate clients, especially, are always prepared. So when they have these episodic incidents, whether it’s a subpoena, third-party or a subpoena, when they have these incidents, we’re already prepared from a planning perspective to make that as flat as possible and as cost-certain as possible. Mouna, I think we can go to the next slide.
Mark MacDonald
Thanks, Sam. If you’re a builder or an attorney representing builders with federal contracts and critical infrastructure projects, especially with government agencies, then you’ve probably heard of CMMC. This will be its own webinar at some point in the future. But again, we’re going to touch just briefly on this. If that is your world, if this is your concern, and this is new as of October, I think it was in 2024, then there are certain types of data you have to be cautious of and make sure that your vendors and your law firms are compliant. And what does that mean? It means that if there are designs for any of the things that come under this, shipping, power plants, that might be a naval yard, whatever it might be, that is encrypted at rest. The vendors and the sites we’re using to process and review that data are locked down to the nth degree. As I talk, I hope you were reading that slide, but Mouna, let’s move to the next one really quickly. As we do that, I’ll also note that construction companies are not unlike most other corporations. Data has grown explosively and continues to grow explosively. If you rolled out Copilot, you know what I’m talking about. It becomes exponential to data growth. On an everyday basis, we’re creating tons and tons and tons of data. And so, if you need to abide by the CMMC, it really all kind of falls back towards data classification. This is something that lots of companies, if not all, are currently struggling with. How do we take unstructured data that lives inside the corporation and categorize it for highly sensitive things that fall under the CMMC? Again, that’s another discussion. But keep in the back of your mind that data classification is ultimately the holy grail. It’s the holy grail for eDiscovery, finding your sensitive data, protecting your sensitive data, and what we might be able to do with AI tools down the road. Then again, how can AI tools come back and feed the effort to classify this data? So, let’s move on. Sam, I saw you smile. Do you want to add something there?
Sam Morgan
I just love your excitement about it. It’s just one of those things on the floor. You can go ahead and go to the next slide, Mouna. Issues that are on the forefront of folks’ minds right now, and it all spills right into the need for what we’re talking about is this eDiscovery blueprint. Once we have a really true understanding of your end goal, we will start our initial conversations with our larger clients. Once we understand why our end goals, we can work backward from there and then create a blueprint or a roadmap. For the construction industry, we’ll use blueprints just like when you’re going to build a power plant, or you’re going to build a building, or you’re going to build a house, the first thing you need is a set of blueprints. Those are going to have to be vetted and approved, as well as everything like that. What it does is that, as I alluded to earlier in the conversation here, it takes the angst out of those episodic events because we already have standard operating procedures around how we react to a subpoena or how we’re going to produce data or how we’re going to have our outside counsel review data within our environments and things like that. And what it does is create a streamlined process; everyone knows that once this comes in, it’s just a matter of course; it’s what we do. At HaystackID, that’s one of the most important steps of any engagement. The goal is always cost reduction. But how do we get there? That is very theoretical. But we actually talk about the use of different tools. We want to understand what the data sources are. We want to understand the implications of new technology like wearable devices and things like that, how they will be subject to discovery, and all these other areas that really touch our industry, as Mark alluded to. The blueprint is going to be your most important starting document going forward. And that’ll be as kind of you. I’ll not only say Holy Grail, as Mark said, but that’ll be your manifesto for success.
Mark MacDonald
Well said, Sam. I couldn’t agree more. Call us vendors or service providers, but we’re the group of people entrusted to help manage your data through the legal process. We like to talk about our bells and whistles, our firepower, our talent, and our people. At HaystackID, we’ve got our needles, like Esther and Young, that help sew this fabric together. But the question is, why do we do what we do? Why do we have all these tools? Why do we have all these resources? The answer to the why question is to help you. It’s to be your voice of reason. The experts at this can consult with you or outside counsel. And again, whether it’s one case or we’re helping you climb the eDiscovery maturity level to level five, at that point, you’re totally in control of your data, whether it’s internal systems, external systems, or some marriage of both. Again, we talk about this repeatable process, eDiscovery, and forensics. The definition of computer forensics or forensics in general is a repeat process, and we get the same results all the time. That’s what our goal is with our clients. Again, one case, many cases over multiple outside counsels. How do we touch the data? Who are the people we go talk to? When you hire, when you’re an owner of a project, and you go and look for your general contractors, you’re hiring the logo and the talent behind that logo. And that’s what HaystackID has developed. We develop a team that has eDiscovery project managers and eDiscovery consultants internally who all have construction expertise. So when you come on board and talk to us, we get on the line. We start to talk about your specific case and need; you will be dealing with a bunch of people, not just one rock star that we sell, but a whole team underneath that can continuously support your efforts and goals, drive efficiency, and reduce costs. And that’s what we’re here for. Sam, again, thanks very much. And look, I want to go back and mention one other thing about the complexity of data, too. We keep seeing different things all the time. I wanted to make a point about something we’ve been seeing a lot of recently, which is video. Video is coming to a site near you, whether it’s a proactive measure or something that’s being enforced by the insurance companies because there’s leakage of materials, trucks disappearing, or for, hey, where are people on the site? Where are they spending their time right now? Wearable devices. Our friends at Pimloc are doing so much with video. So we talk about our partners by name because we also put them in front of our clients. So, there is more to come on all that stuff as well. But I really wanted to drive home the point that without a plan, it’s just a dream. The same is true for eDiscovery, as it is for erecting a building, a bridge, a tunnel, a power plant, or whatever it might be. Start with a plan and move forward from there. So, I just want to drill into that.
Sam Morgan
Let’s start getting to the stars of the show.
Mark MacDonald
We’re going to let the stars speak now. So Sam and I will take a little bit of a backseat. We’ll continue to prompt both of you as we go through the rest of the slide deck here, which is only two more slides, and we’ll probably spend 30 minutes on it. But Esther, you were a former client of HaystackID. We came to you with proofing and vetting and wrote white papers based on the active cases in your previous life. And you’ve brought all that experience here. Our listeners, I’m sure if they could raise their hand right now, want to understand what’s fact, what’s fiction, and how they can use this right now. Where are other companies using it? Where does it make a difference in construction? Let’s start down that path. And welcome.
Esther Birnbaum
Thanks. Yeah. Just to touch a little bit back about what you were saying regarding blueprints. I think blueprints and processes are really important in discovery. A lot of the similarities will run across different matters, different cases, and different industries. However, understanding a specific industry like construction to apply AI processes is really interesting. When Mark and Sam came to me, I thought it was great that HaystackID is starting to create these tailored approaches for different industries. And there’s so much that we can do with AI. And we’re starting to think outside the box about different applications. We’ve done a lot of testing on things like relevancy review and privilege review. But there are just so many different use cases. The ones in construction that could be really interesting and out of the box are analyzing images and video, as Mark referenced. And maybe in one of our next sessions, we can do a deep dive into some of what we’ve seen there because some of the results are fascinating. In terms of basic automating first-level review with GenAI, we’ve seen incredible results that far surpass our results with human review and TAR review. And we’re seeing a lot of companies really start to use it, whether it’s using it instead of doing a human review or using it as an efficiency tool with first-level review to save time and money; we’re really seeing a lot of engagement and so many different use cases. The same goes for privilege and privilege log creation. I remember creating Excel spreadsheets with priv log entries that were hundreds of thousands of lines with drop-downs, and it was just a disaster when we manually edited email addresses. And they were terrible. And now we’re at a time where, okay, we set up a model to run privilege; it identifies privilege documents. And it generates a priv log narrative. And the ones I’ve seen have been really great. They don’t need that much work. So we’re really just moving into this whole new world where there are so many different applications of GenAI. We’re starting to explore specific use cases with construction, and I think it’s exciting.
Mark MacDonald
At this point, I think most people are familiar with TAR, TAR models, and whatever else they might be. When we start with a seed set, we qualify, validate, and QC that seed set with an expert or two. It’s usually 500 documents, but depending on the population size, it might be a thousand documents. How does it start with a GenAI model? What are the similarities/differences?
Young Yu
It’s a very similar approach. When you’re testing your prompt criteria, you really want examples of documents responsive to each of your issues or your buckets, your categories, whatever you want to call it there. But you also want to test against non-responsive docs; you don’t want to lean too far the other way. In terms of sample size, everybody is mixed about this. If you come from a math perspective, your sample, more is better. But understanding that cost and timing are considerations, I think the general methodology is a couple of hundred docs, 500 docs, or a thousand docs, whatever that may be. The validation at the end, especially for GenAI where precedent really hasn’t been set, I think the larger that validation sample, the better. It doesn’t hurt to streamline the process to set up an active learning model, take the GenAI predictions, and see how your documents rank in a more traditional TAR 2.0. Again, I think precedence is going to be set this year, if not sooner rather than later. But I mirror Esther’s initial statement: We’re very skeptical of going into GenAI’s returns, and we’re very surprised by the results. And I do think it’s an art here in terms of generating that prompt criteria, figuring out what legal language works, what doesn’t, and where you need specificity or where things are too specific. Tailoring towards a particular industry, such as construction, provides context around words that are frequently used differently for construction. Then, just understand what you are looking for. You’re really looking for the right question to ask. And that whole prompt iteration, the criteria iteration, is what you’re doing there. For the first level, I think it’s really important that you have someone who knows the case law and your document population. We can talk about cost and efficiencies here. But if you have a million documents to throw in, if we suspect that these are all in your date range for your relevant custodians, how many are truly just junk? In terms of templating and making SLPs, it’s easy enough to come up with search terms relevant to your matter. Building some search terms that kick out non-responsive documents might be easier. The popular one we see and advocate for is just searching for the term unsubscribe. How many mass notifications do you get from advertisers or marketing companies? And if you can kick those out of your population, you’ve already just reduced the corpus of documents you intend to review. The other big search term proponent of this is privilege. Everybody has their standard privilege list. We’ve seen some as short as five terms and some as long as 5,000 terms. There will always be a limiting factor in terms of the number of terms you want to run. But if you’re searching for privileged and confidential as a term, it will hit on every footer. How do you escape that? You can exclude certain footers; you run against the wall there. But in terms of GenAI, that might be the easiest way to go. Even if you do run search terms against your responsive population if you feed those into GenAI for indicators of privilege, and then anything that comes back is not privilege where you have a privilege search term that you’re almost positive is supposed to be a privilege, you can just QC against that. Now, there is some legwork up front, where you’re identifying some of the individuals that generate privilege and potentially identifying those who break privilege. But it’s a model where the determination is made, and you’re getting feedback. So, with human review, you get a breadcrumb trail of tags. This document is privileged; we think it’s privileged for this reason, whether it’s attorney-client, work product, or some other designation. With GenAI, you’re getting the rationale or the reasoning behind why a document was determined to be privileged or not. And it’s helpful when attorneys go through this to look and see, hey, it’s consistently correct here, but it’s consistently missing on this. It becomes easier to identify those pockets of areas. The same is true for first-level reviews.
Sam Morgan
This might be very useful and helpful for the audience, and this is something both your and Esther’s expertise can help a lot of folks with, just a quick little differentiation between typical traditional search terms and prompting. Because they’re used interspersed interchangeably in a way, but same time, just help us out with that also.
Young Yu
It’s funny; you can have search terms, and then you can do an active learning review. So let’s say you start with five million documents, you run your search terms, and you end up with a million documents, and that’s family-inclusive. Then you go through an active learning review workflow, and of that million documents, 300,000 are in response to family-inclusive. So getting from five million to one million is great, but if you look at it lockstep, your search terms limit you down to a million, but there are only 300,000 response documents or 30%. So, while that is great cost savings, how much time do you spend arguing over search terms? What search terms are over-inclusive, and what may be under-inclusive? That’s the other side of the fence here. When looking at search terms, there is an implied understanding that you are leaving things behind. There are 400,000 that you’re not going to review because they were not search terms, and there are things there that you will probably not know about because they’re not search terms. When you speak to GenAI, whether you’re applying search terms or not, it’s much like active learning in that regard; if you were to throw five million documents into active learning and just code documents, it doesn’t really matter whether it’s a search term hit or not, you’ll get to a responsive set. It’s a matter of how many documents you need to review to train the model. With GenAI, the model is already learned; it’s a large language model with an understanding of context and comprehension. It’s just finding the right questions to ask so it’ll pull the right documents for you. In terms of cost savings and bifurcating workflows, you can go either way. What will be exciting and something we’ll be seeing soon is generative AI for ECA in lieu of search terms. We are providing broader baseline criteria for responsiveness, and the goal here is to say, okay, I want to hit a very high level of recall, so all the responsive documents grab as many as we can, understanding that there will be a precision trade-off. So it’s similar to something responsive; we want to pull it back just to take a walk. When we play with that, we’ll certainly be able to provide some case studies or white papers or some qualifying metrics around it. But it will be interesting to see how that plays out because, with precedents up in the air right now, everybody’s waiting for someone to go first and hoping that the person who goes first isn’t going to mess it up for all of us.
Esther Birnbaum
I think the key point that Young made that I think is just really important is that the way you approach a review that’s leveraging GenAI is going to be a little bit different than the way you approach a normal review because it’s really important to understand what your data set is. So, in pre-data culling, understanding what your data is to query goes back to what data has been collected. Prompting and discovery: it’s not very difficult; it’s pretty simple because you’re basically trying to speak in plain language to prompt. And there are nuances. But we saw incredible results using just the issue tags from a review protocol that I ran, and we got incredible spurs for recall and precision without engineering the prompts much. What we did learn is that you want to keep it simple. So if you’re looking at a specific custodian document, you don’t need to say in this custodian’s document, or you can tailor it much more specifically when you take a more holistic look at a matter and the substantive background and the case law, etc., to be able to tailor prompts. We see that subject matter experts, discovery experts, etc., will do a lot more work upfront, but it all translates to incredible cost savings and efficiency downstream.
Young Yu
The biggest gain here is really going to be time, well, time, and let’s call it management. So if you have a very large review, or even, let’s say, it’s not a large review, but you’re under a time crunch, your traditional active learning workflow would work, but how many reviewers are you staffing to meet your deadline? Manage 50 contractor reviewers, in-house or outside counsel reviewers, and get them to make consistent coding calls. Call 10 of your friends and ask them where you want to go for lunch; I’m sure you’ll get 10 different answers. You don’t have that with GenAI. With the benchmarking here, you’ll see some numbers thrown out by providers, but one I can quote is Relativity. We spoke to them yesterday, and we have 300,000 documents a day. You submit that, you send it, and you get your returns. And that’s a day’s worth of work. How many reviewers would you need to get that return? These pipelines will broaden and widen as GenAI evolves and data centers evolve to meet needs in terms of bandwidth and pipeline. But that 300,000 documents that you send early on, once you have a final passable prompt, you can poke and stab at it all you want by looking at the returns, the explanations, the considerations that it’s factoring and the rationales or the reasoning that it’s providing back. So, I think that the feedback loop that GenAI is providing you is probably more insightful than what human review would do because you are getting summaries of documents. You’re getting what is responsive because of this, which you typically don’t get unless you sit with each reviewer individually and ask that line of questioning. For very simple use cases, as Esther stated, we simply took the issues as written on the coding protocol, copy-pasted them as individual issues, and tested them against a sample of 500 documents. Again, Esther came to us with a data set that was completely human-coded through active learning. So, we did have calls, and the data set was thoroughly reviewed. That’s not to say that the human review was infallible, but we understand the human propensity to disagree. We needed a baseline, and that was our baseline; the human review coding was our ground truth. In the sample of about 500 documents, 150 were not responsive, and documents encompassed at least a few for every one of these issues, and we did have nine or 10. And just copy-pasting 85% recall, I think it was like 90% or 92% precision. That really means that out of all the responsive documents identified by reviewers, GenAI’s first pass was able to identify 85% of them just copy-paste. When you look at the language used, how much of that language is just legal talk or filler space? There are terms that we use every day, like internal communications. That might be apparent to us, but to GenAI, maybe not so much. When you have third parties that you inherently know at your company or your outside counsel that you inherently know, you need to provide that level of specificity. There also is a distinction when you’re reading this, reading those document requests, or generating these coding protocols where you say communications between this person and that, but that does encompass attachments. When you use a term like communications, if GenAI comes across a document that’s not communication, it will say this is not communication. But if you intend to say any document from A to B, that’s the language nuance you need to be wary of before you start prompting the system. We worked with Esther to generate a second pass here. And we saw gains. In the way we decided which issues to revise, we tracked each issue individually against the human coding. So when you run GenAI, you can go through a couple of different workflows here. You can do responsiveness in one shot and just copy all your criteria or type all your criteria into one text box. Or you can do issues based on where you’re putting each criterion, itemizing it out, and the system will tag those documents for you. We have a baseline to compare against. And when we go issue by issue versus human coding, we see where the recall is less than optimal. Call that 65%/75%, anything less than that. We worked with Esther to rework the language there for those criteria. So, in our second pass, we gained 5% recall, so we got up to 90% recall. So it is more accurate there. We lost, I think, 3% precision, but we weren’t really concerned there. Three percent, when you’re above 85%, is not really a huge indication of process lost or documents lost. For our last run, we decided to take the best working versions of each issue and throw that up against the sample of 500, the same sample set. And we ended up, I think, with 92% recall and 89% or 90% precision. We were happy with that. How much time did that take if we were to condense that down, Esther? Would you say four and a half hours of billable time?
Esther Birnbaum
Sorry about the background noise. Fittingly, there’s construction in my building right now.
Mark MacDonald
You can’t make it up.
Esther Birnbaum
Yeah, I mean, the process for me to work on the prompts really involved understanding where they were hitting appropriately and where they were not. However, once I better understood how GenAI works, I realized that it was a few hours of work. But as Young said, the results we saw were incredible. And this was not in any way an easy review. It was financial services, they’re very detailed. And I think my response after we ran the full set was, “I never want to do a human first-level review again.” I felt pretty strongly about our results.
Mark MacDonald
In this example that Young and Esther are talking about, Esther, how many documents were in the population so the audience has an idea?
Young Yu
There were about 150,000 documents, Mark.
Mark MacDonald
You started with that after duplication and all the other tricks you used to reduce the data size.
Young Yu
That’s right.
Mark MacDonald
By the way, your unsubscribe trick will go right next to my domain analysis trick. That’s excellent, Young. I hadn’t thought about that one before. That’s a good one. One thing that Young said that people might not have caught is that we are using a blend of a TAR 2.0 review and GenAI right now. If a customer were to come to us right now, we’re not doing a full-blown GenAI review. Is that correct?
Young Yu
I mean, it really depends on that sample. If you come to me with two million documents, I’m probably going to give you 1,500 documents to code and run through GenAI.
Mark MacDonald
Like a standard seed set, just like we would with a TAR reveal?
Young Yu
Right. Right. Now, let’s say you come back and tell me that 80% of the 1,500 documents we sent you to code are responsive. We might just throw everything into GenAI because your responsiveness rate is through the roof here. But if you come back and 10%, 15% of your population is responsive, I’m not sure that that will be the best workflow to throw all those documents through GenAI. If the coding is good and true for that 1,500, let’s say you’re at 15% responsive docs. We would use the coding that you’ve done to train an active learning model. We’d have baseline scoring. We would start feeding documents in through GenAI with a workable or passable prompt. Sampling not only highly ranked documents, but we’d also choose representative scores to suss out where responsiveness would end in terms of score. And continue to send documents through GenAI with some human supervision feedback loop with counsel saying, “Hey, does this make sense? Does this make sense?” Until we can establish, let’s say, okay, we understand where responsiveness is tailing off in terms of active learning. Let’s get a little conservative by dropping maybe three or four points below and say, okay, we understand it’s 15% responsive; let us send the top 20%/25% of documents through GenAI. And more than likely, that’ll be your fully responsive set. So long as you have representative examples of all your criteria for responsiveness encompassed. That way, we can limit the cost there. But again, with early adoption here, it might just be if you want to see how this works, we might just send the first; let’s say the budget is not a problem for anybody, or you have deep pockets.
Mark MacDonald
Still looking for that client, by the way, but okay.
Young Yu
Right, right. The data science and analytics teams would love to send it all. And say, “Hey, look, here are the results; what do you think of the results?” We understand that for everyday business, that’s not a possibility, and we will look for ways to limit your data. But again, it goes to say like, hey, if you have a blueprint, if you have SOPs in place that you’re dropping in, or let’s say even if you have a playbook where you understand that every data set that comes in, you’re going to handle on a specific matter. Now, it can be purpose-built, so whether it’s for a labor dispute or another type of litigation, you can repurpose or, let’s say, re-tailor these different playbooks, and as long as you can get something in place that is uniform and a repeatable process. And especially from, let’s say, an end client role, if you are that company that’s farming work out to outside counsel and vendors, if you have uniformity across the way that your counsel and your vendors are assessing and reviewing data, you’re going to come to a repeatable process. It is no surprise that you can estimate the cost and delivery schedule. And it’s like litigation readiness; you hope for the best and plan for the worst.
Esther Birnbaum
Yeah, I think the key point is that with each review or client, we will tailor our workflow because every matter is different. The process with GenAI is changing how we approach these reviews. So, GenAI is obviously the new hot tool to talk about, but we have proven and practiced other AI tools that we can use together with GenAI to create the most efficiencies. And Young and his team are the best at putting those together. It’s almost like a puzzle about which workflow and what tools are going to work best for each matter.
Mark MacDonald
Construction reference: We’ve got a toolkit ready for whatever the job might be, and we have the operators to run it.
Esther Birnbaum
The amount of construction jokes you make is a little bit worrisome to me, but it works here.
Young Yu
It is true, though. It is a toolkit. You want to find the best tool for your problem. Understanding the nuances of large language models will be important in the coming days. They are purpose-driven as well. You’re not using the same large language model (LLM) to do CAD design as you are to write code as you are to, let’s say, do a video or audio analysis. So I see it moving in the future here, having someone on staff say, “Hey, this works for this, or this works for that.” It doesn’t have to be data science level understanding, but understanding that these LLMs are also purpose-built and driven. What works best for you for this type of matter or this type of data may differ from area to area, practice area to practice area, and maybe even jurisdiction to jurisdiction.
Esther Birnbaum
The other thing that Young touched on was the direction we’re moving toward, where we will be leveraging GenAI with ECA. We’re going to start the process of identifying and classifying documents earlier. And I think in the next six months, we’re really going to see that adopted. One of the big use cases is that it’s an early case assessment workflow, but for third-party productions, parsing through them is the biggest use case we’re seeing in GenAI.
Mark MacDonald
I’m sorry, Esther. It’s just epic that the jackhammer or whatever drill is going off in the background. Welcome to living in a New York City apartment. You can’t control that stuff. We’ve got nine minutes left. Let me ask a couple of simple layman’s questions that people who have not yet dipped into the world of AI might be curious about. How safe is their data? What can we use it on? Esther, I was able to see a beta of the ECA tool that you talked about because we were having that conversation, like I said, on Friday, a quarter of a million documents, and still so much to talk about. Attorneys will battle over keywords, having not yet seen any data, and that’s the position that this particular attorney is in, so to get to the next level in how we do what we do, being able to run those documents through an ECA tool that’s not really ECA because we’ve already got the data in Relativity at this point. It was fascinating to see the output from your sample set. Is this the end of the keyword? Is this the death of the keyword? You don’t have to answer that on the webinar; I won’t put you on the spot. But it’s intriguing. So, is the data safe? What can we use it on? The example we discussed with that particular attorney is that she’s concerned about opposing party productions. She thinks that they’re going to get dumped on by the opposing party. She gave us the order: don’t load up any opposing party productions yet into Relativity until we have some type of game plan for that. Now again, if we’re doing the doc review or someone’s doing the doc review and we’re getting all this great coding from the TAR review that we’re doing, how do we leverage that against the inbound productions? Again, we’re at eight minutes left, so these are the seeds for the next series of webinars we will be doing. But maybe quick answers to those.
Esther Birnbaum
So, in terms of, is this the death of search terms? Lawyers love precedent; that’s what our entire industry is based on. And I think a lot of what we’ve done in discovery historically has continued because we’ve done it historically. And we see that changing, and we see AI disrupting. It will disrupt so much of the process that we’ve seen in the past. There will be resistance to saying, “Hey, we have a GenAI tool that can parse through our data at X cost that will give us better results in search terms.” Not everybody will advocate for that, depending on what side of a matter you’re on. Search terms are easy; they are things everybody understands and lawyers understand. It drives me crazy when people who don’t understand discovery will meet and confer and agree to.
Mark MacDonald
Negotiating search terms.
Esther Birnbaum
Negotiating search terms, but as you said, never seeing the data, never knowing what they hit on, and not realizing that you can’t run some terms. And we can talk about that another time. We know that this is a better approach with GenAI. This is going to get us better results. Our ethical duty as lawyers is to make sure we’re doing that. At the same time, there’s going to be pushback because there’s a cost, and search terms don’t really cost much. But you also have to look at the downstream cost of having more accurate data sets that are actually going to a full review. That’s why I think that everyone’s jumping on GenAI for the use cases for opposing-party productions, third-party productions, and that type of thing. And I did it. In my practice, I used a GenAI review to look at a third-party production. And it was like, “This is fascinating.” I was working with our outside counsel. When they saw the work that I was able to do with Young and his team with GenAI, their response was, “I’m shook.” Those were written in emails.
Mark MacDonald
A lot of times, the third-party productions are just ignored because there’s a significant cost to examining those documents, just as there is with the outbound productions that we’re creating.
Esther Birnbaum
And you’re talking about the unsubscribe and the domain analysis, and yeah, we all have these tricks that we use. But with GenAI and its ability to classify data at such a level, we’re not going to have to do that; it won’t be a manual process. You can identify junk email, these email lists, and things like that because you’re going to be able to see a picture of your data with ECA tools. And we’re not quite there to launch them yet. I think we’re going to start to see it at LegalWeek. But I’ve tested them, and it’s going to blow you away. So this is just the beginning, what we’re talking about, and the beginning is amazing, but I’m like, I can’t even imagine what we’re going to see in six months.
Mark MacDonald
I think about arbitrations and negotiating before discovery ever happens and getting insight into data. And like in the example I keep talking about, can our tools—like the ECA tool that I saw last week—help identify keywords without even needing humans? Can we just say, “Here you go, there are the most populous keywords in this data set?”
Esther Birnbaum
Yeah. There are going to be tools that you can use to see a word map. Every tool we have seen in the past decade has said, “Okay, we can show you communications based on the weight of a line between two or three individuals.” But with GenAI, you’re going to be able to see not only the keyword map but also the topics of conversations between certain individuals on a bigger scale. You’re just going to have an enormous amount of insight into your data. And yeah, I mean, there are going to be a bunch of different ways that’s going to even help tailor to keywords. But I think at that point, you’re going to be like, “I don’t need to because I already know where the documents that I need are.”
Mark MacDonald
Interesting.
Young Yu
Certainly agree with Esther. I do think I want to point out that the user or the driver of the application is going to become very important. So the person who’s prompting the model, the person who’s entering the criteria for that final prompt or putting either in layman’s terms or providing that context to GenAI in terms of that question that’s being asked, that’s going to become a very important factor in, one, I think accuracy and, two, validity. Sometimes, the thought is, and the concern has always been, that where there’s bias introduced into either the model or the methodology, there’s going to be an effect on the outcome. And I don’t think that you can totally remove bias from the training because the models have been trained on what they’ve been trained on, and everybody who’s probing a set of data is looking for something specific. But I do think that the person entering or providing that prompt language, whether it’s an individual or a collaborative effort, the line of questioning there definitely weighs into your returns and the insights you receive back. If you don’t know the question to ask, it will be hard to get an answer because you haven’t asked the question. So, determining the series of questions you want to ask and probing your data for will become very important. That’s going to become a skill set.
Mark MacDonald
Anyone who’s known me for any period of time will have heard this one before. Anyone can buy a Formula One Ferrari, but that does not make you Mario Andretti. Just because the tools exist, you still need this expertise to drive it, train it, and teach others how to use it. Here we are at 1:00 our time, so I’ll be respectful of that. If anybody put a question in that we didn’t get to, I’m so sorry. We will answer you one-on-one with a heartfelt response and introduce you to our experts if needed. But guys, thank you so much for being here. Sam, thank you so much for everything, of course.
Sam Morgan
Esther and Young, thank you.
Mark MacDonald
Yeah, this was very, very helpful, very good. And you know where to find us.
Sam Morgan
Yes, we have tons of topics. For HaystackID’s webinar series, our next will be next week on managing data overload complexity in catastrophic events. I hope some or all of you can join that as well. Thank you all.
Esther Birnbaum
Bye.
Mark MacDonald
Have a great day. Thanks a lot.
Sam Morgan
Okay. Bye-bye.
Expert Panelists
+ Esther Birnbaum
EVP, Legal Data Intelligence, HaystackID
With a robust background in complex litigation and regulatory compliance, Esther brings a wealth of knowledge and practical experience to the table. She uses her unique expertise at the intersection of technology, data, and law to develop best practices and drive innovative workflows across many areas of the business. She enjoys sharing her insights with the wider eDiscovery community and frequently speaks at conferences, webinars, and podcasts on topics related to law and technology.
+ Mark MacDonald, CEDS (Moderator)
VP of Enterprise and Strategic Accounts, HaystackID
Mark is a seasoned eDiscovery professional with over 20 years of experience helping corporate legal departments design and implement effective and defensible ESI management models. He is a certified eDiscovery specialist (CEDS) and a thought leader in the industry, with expertise in legal hold, data breach, digital forensics, advanced analytics, managed review, and eDiscovery playbook development. Mark has managed thousands of matters and served as an expert witness across various industries and types of cases on several occasions. Since January 2023, Mark has been the Vice President of Enterprise and Strategic Accounts at HaystackID, responsible for consulting and supporting clients on best practices and workflows across the entire legal data lifecycle, leveraging HaystackID’s unrivaled blend of customer service, talented professionals, and innovative technologies. Mark’s mission is to ensure that his clients benefit from technology-driven, scalable, and cost-efficient solutions that address their unique ESI-related challenges and goals.
+ Sam Morgan (Moderator)
Director of Legal Solutions, HaystackID
Sam Morgan is a Director of Legal Product Solutions at HaystackID. Harnessing his more than two decades of industry experience, Sam drives immeasurable value for his law firm and enterprise clients. Having worked as a paralegal at leading AMLAW 100 firms, Sam knows firsthand the challenges that in-house teams and eDiscovery practitioners face and provides clients with tailored strategies and solutions to overcome these obstacles like handling ballooning data volumes and a proliferation of data sources. Sam has a proven track record of success for some of HaystackID’s most prominent global corporate and law firm clients. Taking a consultative approach.
+ Young Yu
VP of Advanced Analytics and Strategic Solutions, HaystackID
Young Yu joined HaystackID in 2018 and is currently the Vice President of Advanced Analytics and Strategic Solutions. Prior to his current role, Yu was the Director of Advanced Analytics and Strategic Solutions at HaystackID. In this role, Young was the primary strategic and operational adviser to HaystackID clients in matters relating to the planning, execution, and management of eDiscovery activities.
About HaystackID®
HaystackID solves complex data challenges related to legal, compliance, regulatory, and cyber events. Core offerings include Global Advisory, Data Discovery Intelligence, HaystackID Core® Platform, and AI-enhanced Global Managed Review powered by its proprietary platform, ReviewRight®. Repeatedly recognized as one of the world’s most trusted legal industry providers by prestigious publishers such as Chambers, Gartner, IDC, and Legaltech News, HaystackID implements innovative cyber discovery, enterprise solutions, and legal and compliance offerings to leading companies and legal practices around the world. HaystackID offers highly curated and customized offerings while prioritizing security, privacy, and integrity. For more information about how HaystackID can help solve unique legal enterprise needs, please visit HaystackID.com.
Assisted by GAI and LLM technologies.
Source: HaystackID