Safe and Effective Use of GenAI in Review: It’s Dangerous To Go Alone

Generative AI (GenAI) represents the latest advancement in the ongoing struggle against the ever-growing volume and complexity of ESI discovery and legal document review.  In simple terms, GenAI is a type of artificial intelligence that can generate new content, like text, based on prompts from a user.  In this context, we are concerned specifically with large language models (LLMs), which are trained on large collections of text documents to learn linguistic patterns and features.  This makes them good at reviewing text and answering questions about it, which in turn makes them powerful tools to aid in legal document review.

The more powerful a tool, however, the greater the potential harm from its inadvertent misuse – as any of the attorneys sanctioned in the past year for citing imaginary cases can attest.  Attorneys have an ethical duty of technology competence that requires staying abreast of the benefits and risks of new practice technology.  How can attorneys fulfill their duty of technology competence with regard to leveraging GenAI tools to aid in document review?  As Legend of Zelda games have famously told us since 1986: “It’s dangerous to go alone!”

Curative Assistance

Over the past twelve years, forty states have adopted some form of technology competence requirement for lawyers.¹ Although this technology competence requirement was originally spurred in large part by the rapid rise of ESI and eDiscovery, it is not limited to just that area², encompassing data protection, communications technologies, practice technologies, client technologies, and more.  Without question, GenAI technologies and GenAI-assisted review (GAR) workflows based on them must now be added to that list, as proliferation and adoption of both accelerates.

Thankfully, recognizing the practical limits of how technologically adept a practicing lawyer can be expected to become, many states allow or encourage the fulfillment of their technology competence requirements through the “curative assistance” of qualified experts (e.g., California, Florida, Washington).  Knowing when and who to call for help can be as effective as learning the answers oneself.  As in the early days of TAR, practitioners today will benefit from working closely with experts to explore the potential applications and benefits of these new GenAI options.

Expertise Required for Safe and Effective Use

As with TAR, safely and effectively leveraging GAR requires review management expertise and legal domain expertise, but it also requires all new prompt design expertise for interfacing with the new GenAI tools:

• Review Management Expertise
  • There are numerous points in the review process at which GAR may be leveraged, from prioritization to classification to quality control, and in more complex workflows GenAI tools could also be used in a layered approach in conjunction with TAR tools (e.g., to help identify the best training materials) or CAL tools (e.g., to reduce the population prior to GAR). Figuring out the best combination of tools and techniques to deploy in a particular matter, how best to deploy them, and how best ensure quality control requires expertise with the range of possible workflows and potential challenges.

• Legal Domain Expertise
  • In addition to having expertise in document review itself, expertise in the relevant legal domain is also crucial to any document review effort – especially one leveraging GAR. Beyond the technology and technique, document review is making determinations regarding the factual and legal significance of each documents’ content.  These determinations are guided by a complex protocol that factors in the legal, corporate, and industry context.  Having expertise with the relevant matter type, industry, and organization type is necessary to effectively implement such a protocol, especially when it must be “translated” into a form suitable for use with a GenAI tool.

• Iterative Prompting Expertise
  • Most essential to leveraging GAR safely and effectively is having expertise with how to design, test, and iterate prompts for these new GenAI tools. In a way, GenAI tools are like the faeries or djinn in cautionary folk tales: they can grant you powerful boons, but the exact wording of the request, matters a great deal.  These are tools you must talk to, and how you talk to them makes a world of difference in what they’ll say to you and how long it will take them to say it.  How should you phrase each question to learn what you need?  Which questions should be combined and which should stand alone?  Which order should you ask them in?

Key Takeaways

The more powerful a tool, however, the greater the potential harm from its inadvertent misuse, and new GenAI tools can be very powerful indeed.  Attorneys have an ethical duty of technology competence that requires staying abreast of the benefits and risks of new practice technology, including GenAI, which they can and should fill through curative assistance from qualified experts.

As with TAR, safely and effectively leveraging GAR requires review management expertise and legal domain expertise, but it also requires all new prompt design expertise for interfacing with the new GenAI tools.  These are tools you must talk to, and how you talk to them makes a world of difference in what they’ll say to you and how long it will take them to say it.

¹ Robert Ambrogi, Tech Competence, LawSites, https://www.lawsitesblog.com/tech-competence (last visited July 2, 2021).

² Steven M. Puiszis, Perspective: Technology Brings a New Definition of Competency, Bloomberg Law, https://news.bloomberglaw.com/business-and-practice/perspective-technology-brings-a-new-definition-of-competency (Apr. 12, 2016).

‍