Reported The Invented Source Costly
Six court cases, correctly formatted, complete with judges. None of them existed.
A New York lawyer filed a brief citing six precedents ChatGPT had invented — with plausible names, docket numbers and quoted opinions. He then asked the same chatbot whether the cases were real. It said yes.
Asking the model whether its own citation is real is not checking. Open the source.
Roberto Mata was suing the airline Avianca over an injury on a flight. His lawyers at Levidow, Levidow & Oberman needed to argue against the airline's motion to dismiss, so Steven Schwartz went looking for supporting precedent.
He used ChatGPT. It gave him six cases. They had convincing names, real-looking docket numbers, named courts, and passages of quoted judicial reasoning. He put them in the brief.
None of the six existed.
The detail that makes this more than carelessness is what happened when doubt first surfaced. Schwartz went back to ChatGPT and asked whether the cases were real. It told him they were. He asked it to produce the full text of one. It produced one.
Opposing counsel could not find the cases. Neither could the court. On 22 June 2023, Judge P. Kevin Castel of the Southern District of New York ordered Schwartz, his colleague Peter LoDuca and their firm to pay $5,000 between them, finding bad faith — not for using the tool, but for the initial denials and for filing work nobody had checked against an actual legal database.
It is tempting to read this as a story about one lawyer being lazy. The more uncomfortable reading is that the verification step he performed felt like verification. He had a doubt, he checked, and the check confirmed the answer. The system was perfectly willing to vouch for its own invention, in the same confident register it had used to produce it.
A citation is the easiest thing in the world for a language model to fabricate convincingly, because the format is rigid and the content is the part nobody reads closely.