Short answer: to understand a concept, yes. To cite case law, no, not on its own. ChatGPT, Claude and Gemini produce references that look accurate but do not exist, a risk already flagged by French courts in 2025. A specialised legal AI like Plato avoids it: every reference points to a real, verifiable decision.
A hallucination is information generated by an AI that looks credible but is false: an invented ruling, a docket number that matches nothing, a citation attributed to a decision that does not exist. The model does not "lie"; it statistically completes a plausible answer, with no verifiable basis in truth.
In law the problem is acute: a false reference has the exact appearance of a real one (court, date, number), which makes it hard to spot without systematic verification.
Yes, and this is no longer theoretical. Courts have observed it in recent decisions. The table below summarises the documented cases in France and abroad.
| Decision | Date | What happened | Penalty |
|---|---|---|---|
| Grenoble Administrative Court (no. 2509827, no. 2512468) | Dec. 2025 | Fictitious case-law references from a generative AI, in the pleadings of self-represented litigants | None (observation) |
| Orléans Administrative Court (no. 2506461) | 29 Dec. 2025 | About fifteen entirely false references cited by a lawyer | Warning, no financial penalty |
| Mata v. Avianca (SDNY, United States) | June 2023 | Six decisions invented by ChatGPT in a brief | $5,000 fine |
| Johnson v. Dunn (N.D. Alabama, United States) | July 2025 | False AI-generated citations | Lawyer removed from the proceedings |
In France, judges have so far taken an educational stance: they note the phenomenon and warn, without imposing penalties. Nothing guarantees this tolerance will last.
Beyond the credibility lost before the court, the lawyer engages their duty of competence. Using a tool without knowing it can hallucinate, then producing its output without checking it, amounts to a professional breach. Responsibility remains the lawyer's, not the tool's: technology does not remove the obligation to verify your sources.
Three simple rules cut the risk to almost zero.
A general-purpose AI draws on broad, unverified knowledge. A vertical legal tool works from a base of identified decisions and ties each statement to its source. For valuing a personal injury case or an employment dispute, where every amount must be defensible, the second approach avoids hallucination by design.
Plato, the only AI specialised in personal injury, is built on this principle: every amount is tied to real decisions from its base of more than 84,000 quantified rulings, and remains traceable back to the exhibit it comes from. Another practical difference: where a general-purpose AI such as Claude caps the number of documents (in the region of 20 PDFs), Plato handles a whole case file, often several hundred exhibits. For the detail of the differences, see our comparison Plato versus ChatGPT, Claude and Gemini and the page Plato for personal injury.
Can ChatGPT cite case law that does not exist?
Yes. It generates plausible references from probabilities, with no basis in truth. French courts (Grenoble, Orléans) noted false AI-generated references in December 2025.
Has a lawyer already been penalised for this?
In France, not yet: judges have warned without penalising. In the United States, yes: in Mata v. Avianca (2023), two lawyers were fined $5,000.
How can you tell whether an AI invented a decision?
By checking the reference against a primary source (Légifrance, Judilibre or the court's own database). A reference that cannot be found should raise a flag; but Légifrance lists only a selection of trial-court decisions and Judilibre covers Courts of Appeal only since April 2022, so "not found" does not always mean "invented". When in doubt, a reference you cannot verify should not be cited.
Does a specialised legal AI hallucinate too?
The risk drops sharply when the tool ties its answers to real, verifiable decisions instead of generating from memory. That is the principle of vertical tools such as Plato.
Can you still use ChatGPT for law?
Yes, to explain a concept, structure an argument or rephrase a text. Not to produce references without checking them one by one.
Is the lawyer liable for the AI's errors?
Yes. Liability remains the lawyer's, under their duty of competence. Using a tool does not exempt them from verifying their sources.