A language model produces the most probable text, not the true one. Most of the time the two coincide closely enough. With citations they do not, and the reason is structural: the format of a case reference is highly regular, so generating a plausible one is easy, and nothing in the generated text tells the real one from the invented one.
The Italian case
The Court of Florence, business division, in an order of 14 March 2025, faced a filing citing judgments that did not exist. Counsel stated that the references came from research carried out by an assistant using ChatGPT, and that the system produced those references again on a second query.
The question the court had to decide was whether that amounted to aggravated liability for vexatious litigation under Article 96 of the Italian code of civil procedureopens in a new tab. The answer was no: an isolated, good faith error does not automatically produce procedural liability.
The part that matters, though, sits next to it. The court wrote that these tools can be a genuine support for research and for drafting, but that using them does not replace the professional’s duty to check and verify the information. That is the sentence which, in a less fortunate case, turns the decision the other way.
Why asking a second time is useless
The most instructive detail is that the citation was confirmed on a second query. Asking a model “are you sure?” is not verification: the answer comes from the same process that produced the error, and consistency between two answers is the thing the system is best at.
The only verification that counts is outside: open the database, look up the reference, read the headnote. Thirty seconds per citation, and there is no shortcut that removes them.
Not a case, a series
The first case anywhere was in New York in June 2023: two lawyers and their firm sanctioned five thousand dollars by the federal judge for filing a brief with six ChatGPT-generated citations that did not exist. The prevailing reaction in Italy at the time was that it could not happen here.
Three years on, the known Italian cases number at least five: alongside Florence, decisions in Turin, Latina, Milan and Syracuse are cited. In one of them the costs order against the party that filed reached the order of thirty thousand euro, which is a different thing from a telling-off.
The series teaches more than any single case, because the mechanism repeats almost identically. Nearly always the research had been done by somebody else. Nearly always the person who signed did not know a model had been used. And nearly always nobody opened the citations before filing. Those are three separate failures closing in three separate ways (a written rule about which tools are used, an internal duty to declare when research was done with a model, and a check before filing), and none of them asks anyone to give up the tool.
The second kind of hallucination, the one that gets through
The invented judgment is the one that makes the papers because it is easy to expose: you search the number and it is not there. There are two other forms, and they are worse precisely because they survive a distracted check.
The second is the real citation, wrongly attributed: the number exists, the division is right, the date matches, and the proposition the model has it stating is one that ruling never stated. It happened in Florence too, where the numbers pointed at Supreme Court decisions whose content had nothing to do with the issue in the case. Anyone verifying by checking that the citation exists lets it through, because the citation does exist.
The third is the superseded source: a provision cited in its pre-reform version, an abandoned line of authority presented as current, a threshold amount from three years ago. It is not even a model error in the strict sense: it reflects what it read, and what it read has a date.
So verification has three steps rather than one. It exists: search the number. It says this: read the holding. It still stands: check whether it has been overtaken. A database does the first; a person does the second and the third, and those are the two that decide whether the filing holds.
Where hallucinations do the most damage
Case citations are what reaches the newspapers, but they are not the only case. A model invents an article number, a date of entry into force, a threshold amount or the version of a rule before an amendment with the same ease. All of them are quick to check and nobody checks them, because they are written with the air of a detail.
The practical rule that holds is simple: anything with a number gets verified at the source, anything that is reasoning gets read with your own head. What you can accept without checking is the form.
How this meets the other half of the problem
Hallucinations and confidentiality look like two separate conversations, and they meet at one point: how the question is framed. Paste the whole file to get some research and you get a vaguer answer, more prone to invention and more dangerous for the data; frame the legal question in three lines and you get a more checkable answer that exposes nobody.
The long version of that argument, with how to research precedent without handing over the file, is in its own piece. What makes a human review real is in another.