When your opposing counsel cites a fake case fabricated by a generative artificial intelligence tool in a court filing—a so-called “hallucination”—should you snitch? At least two courts say yes.
As generative AI increasingly permeates our personal and professional lives, practitioners, courts, and creators face challenges such as hallucinated citations, privilege concerns, and regulatory uncertainty. Yet a single theme emerged during a well-attended Showcase Program at the Business Law Section’s Spring Meeting titled “AI in the Trenches and on the Bench: A Business Law Toolkit for In-House, Firm, and Courtroom”: When using AI, the human element remains indispensable.
The panel discussion, moderated by Paulette Rodríguez López of Crowell & Moring LLP, and featuring Jeffrey Huang of Pilot Company, Alina Lee of Aspire Law, Bradford Newman of Eversheds Sutherland, and the Honorable Richard Platkin of the Commercial Division of the Supreme Court of New York State, explored trends in and actionable guidance on AI’s evolving role in business law practice.
1. Of Humans: Context Matters
Human context and intent are the linchpins of proper AI use in legal practice. “Bad jury instructions, like bad prompt engineering, produce results that we don’t want,” said Judge Platkin.
Huang explained that context is central because generative AI outputs are products of iterative prediction and designed to be sycophantic; they are biased to appear helpful, even if the substance is inaccurate. Unlike human law clerks, Judge Platkin noted, AI models rarely respond “I don’t know,” nor do they systemically apply “learned legal doctrines or review of primary sources.”
Those design features of generative AI can result in hallucinations, including fake cases, inaccurate facts and quotes, and incorrect holdings. According to a 2024 study by researchers at Stanford discussed during the panel, even premium legal AI tools hallucinated between 17 and 33% of the time. Therefore, the panelists suggested that making the user’s goals, constraints, and contexts explicit in the prompts can help reduce hallucination risks and stressed that human-in-the-loop procedural design is essential.
2. By Humans: Building Scaffolding for AI Use
The panel’s discussion of the latest updates in AI jurisprudence offered concrete guideposts for incorporating AI tools in our professional work safely and ethically.
Know AI, Review AI, Disclose AI
Newman explained that lawyers do not need an entirely new ethical framework to begin governing AI use because existing professional responsibility rules already apply. ABA Model Rule of Professional Conduct 1.1 (competence) requires attorneys to understand how the AI tools they use function. Rules 5.1 and 5.3 (supervision) require attorneys to review generative AI outputs as competent attorneys would supervise and review the work of junior lawyers and paralegals. Rule 3.3 (duty of candor) obligates attorneys to disclose their use of AI tools when the applicable court rules require such disclosure and when the clients would reasonably expect to know about such use and the disclosure is necessary for the clients to provide informed consent.
Courts Are Split on Attorney-Client Privilege and Work Product (For Now)
Newman also discussed a court split on whether attorney-client privilege and work-product protections apply to communications involving AI tools. In U.S. v. Heppner, the U.S. District Court for the Southern District of New York reasoned that the AI outputs involved in the case were not protected attorney work products because the client had independently retrieved those outputs without the counsel’s direction, nor were the clients’ prompts privileged because they effectively amounted to disclosure to third parties. In contrast, the U.S. District Court for the Eastern District of Michigan in Warner v. Gilbarco, Inc. ruled that the AI outputs in the case qualified as protected attorney work products because the AI platforms used were better seen as mere tools. However, Newman predicted that this divergence will be short-lived, given that AI use by legal practitioners and pro se litigants is already widespread.
Toward a Federal AI Regulatory Framework?
The patchwork of state AI laws will likely be harmonized under a federalized AI regulatory framework with a federal regulator, Newman argued. Certain foreign jurisdictions such as the European Union, Canada, and Brazil have adopted more comprehensive frameworks. The state-by-state approach the U.S. has taken thus far, he warned, creates uncertainty and significant compliance burdens for businesses.
3. For Humans: More Capability, More Questions
The panel also discussed AI’s potential to expand an individual’s capability, whether a creator or pro se litigant, and novel questions that arise as a result.
Copyright: Human Creativity
“What AI produces on its own is not ownable and not copyrightable,” Lee explained, just as “monkey selfies” were not copyrightable. But while the U.S. Copyright Office concluded in a January 2025 report that current generative AI tools do not “allow humans to have sufficient control, the [Office] specifically [said] that this could change,” Lee said.
She offered illustrative examples of when work involving generative AI is and is not copyrightable.
- Movies and comics with AI parts: The AI-created parts are not copyrightable, but the whole movie or arrangement of AI-created images is copyrightable.
- AI-assisted brainstorming: The output is copyrightable because the human author still creates the output with the AI tool helping the author to generate ideas.
- Visual art modified by AI: An artist who made a drawing and used AI to modify it was able to copyright the product by “specifically disclaiming what AI did” and submitting “the original input and the AI modification,” Lee said.
- Prompts themselves: It may depend on “how creative [the applicant’s] prompt is. If [the applicant’s] prompt is very detailed and includes creativity rather than generic questions, it is more likely the prompt is also copyrightable,” Lee said.
Lee pointed out that despite the U.S. Copyright Office guidance, many questions related to generative AI and intellectual property remain. For example, as AI-assisted coding becomes a mainstay, how will it affect the valuation of codes in M&A?
AI and Pro Se Litigants: Tool or Trap?
Judge Platkin noted that as more pro se litigants submit AI-generated court filings without consulting an attorney, courts and the legal profession face novel challenges as a result. Courts must review mounting volumes of cases without the filter of trained counsel. While AI tools may give nonlawyers better chances to argue on the merits, the self-represented litigants also lack the legal training necessary to identify hallucinations or assess the merits of their argument. Though courts may be lenient toward pro se litigants who submit court filings with hallucinated claims and citations, more courts are imposing sanctions on vexatious pro se litigants, Judge Platkin said.
* * *
Judge Platkin’s statement toward the end of the session captured the spirit of the conversation: “I would not trust AI to make important decisions, but I would trust my decision more if I used AI,” he said. AI may reshape legal practice, but lawyers, judges, and clients will continue to bear responsibility for how AI tools are used. As adoption of AI accelerates across the profession, the discussion suggested that the future of AI in law will depend less on the technology itself than on the human judgment guiding it.
