AI tools developed by humans serve human needs

AI in the trenches is reshaping how lawyers draft pleadings, check citations and manage privileged information, but recent court rulings remind practitioners that the human element remains essential.

Human judgment needed to curb AI hallucinations

During a Business Law Section showcase, a panel of attorneys and a New York Supreme Court justice examined the risks of “hallucinated” citations—cases that never existed but appear in AI‑generated drafts. The discussion highlighted that generative AI often produces confident‑sounding but inaccurate statements because the models are designed to please the user rather than admit uncertainty.

Judge Richard Platkin compared poor jury instructions to bad prompt engineering, noting that both can lead to undesirable outcomes. He warned that AI tools rarely respond with “I don’t know,” unlike junior clerks who might flag a gap. A 2024 Stanford study cited at the forum found that even premium legal AI products generate false references 17 % to 33 % of the time.

Panelists agreed that explicitly stating goals, constraints and context in prompts can lower the chance of hallucinations. The consensus was that a human‑in‑the‑loop approach—where attorneys review and verify AI output—remains the safest workflow.

Existing ethics rules already cover AI use

Jeffrey Huang and Bradford Newman explained that lawyers do not need a brand‑new code of conduct for AI. The ABA Model Rules already require competence (Rule 1.1), supervision (Rules 5.1 and 5.3) and candor (Rule 3.3). Those provisions mean attorneys must understand how their AI tools work, supervise the results as they would a junior associate’s draft, and disclose AI assistance when court rules or client expectations demand it.

Newman noted a split in how courts view attorney‑client privilege for AI‑generated communications. In U.S. v. Heppner, a Southern District of New York decision held that prompts entered by a client were not privileged because they were effectively shared with a third‑party tool. By contrast, the Eastern District of Michigan in Warner v. Gilbarco treated the same outputs as protected work product, reasoning that the AI platform was merely a tool.

In practice, this means a law firm must train staff on both the technical limits of their AI software and the ethical duties that already apply. Failure to do so could expose the firm to malpractice claims if a fabricated case slips through unchecked.

For a pro se litigant, the stakes are different. Judge Platkin warned that self‑representing parties increasingly submit AI‑generated filings that contain false citations.

Ethics remain essential.

From a practical standpoint, the rise of AI tools offers a double‑edged sword for non‑lawyers. On one hand, they gain access to research capabilities previously reserved for trained attorneys; on the other, they lack the expertise to spot inaccuracies, which could undermine their cases.

In everyday terms, a self‑represented defendant using an AI chatbot to draft a motion might think they have a solid argument, only to discover the cited precedent never existed. That misstep could lead to a dismissal or even a contempt finding, illustrating how the technology’s convenience can backfire without proper oversight.

Copyright and the limits of AI‑created works

Alina Lee clarified that AI‑generated content is not automatically copyrightable. The U.S. Copyright Office’s January 2025 report stated that current generative tools do not give humans sufficient control for protection, though the agency left room for future changes. Lee gave examples: a film that includes AI‑generated scenes is still protectable as a whole, while the standalone AI images are not.

When AI assists in brainstorming, the resulting ideas can be copyrighted if a human author shapes the output. Visual art that is modified by AI may also qualify if the artist documents the original input and the AI’s contribution, she said. Even prompts can be protectable if they display a level of creativity beyond generic queries.

These nuances matter for transactions involving code or other digital assets, where valuation hinges on intellectual‑property rights. As AI‑assisted coding becomes routine, parties will need to negotiate who owns the resulting software and how royalties are allocated.

Overall, the panel’s message was clear: AI expands the toolbox for lawyers and litigants, but the responsibility for accuracy, confidentiality and ethical compliance stays with the human user. As the technology evolves, the legal profession will continue to adapt its standards, but the need for skilled judgment will not disappear.