Legal AI Hallucination Rate by Tool: Lexis, Westlaw, Practical Law and GPT-4 Compared
Practitioners exploring AI for lawyers need these numbers before they rely on any research assistant in client work. The Stanford HAI study on legal model hallucinations tested more than 200 legal queries hand-scored by legal experts and defined a hallucination as a response that is either factually incorrect or misgrounded — citing a real source that does not actually support the claim.
This article defines what counts as a hallucination in a legal-research context, walks through how the Stanford benchmark measured rates across 202 expert-scored queries, and presents the numbers tool by tool alongside accuracy and refusal rates so you can compare reliability directly.
It also covers why the tools produce such different results, where the benchmarks fall short, the growing wave of US court sanctions against attorneys who filed fabricated citations, and the concrete steps that keep AI output from entering your brief unchecked.
This article is for information only and is not legal advice.
What a legal AI hallucination is
A legal AI hallucination is defined as a response that is either factually incorrect or misgrounded — where the tool cites a real, existing source that does not actually support the proposition it is offered for. The Stanford study used this two-part definition when scoring more than 200 legal queries, and it captures both the obvious failure and the subtler, more dangerous one.
Factually incorrect vs misgrounded
A factually incorrect answer states something that is simply wrong: the wrong elements for a tort, a repealed statute treated as current, a holding that is the opposite of what the court held. These errors are the kind most people associate with large language model failures, and they are relatively easy to catch with a quick lookup.
A misgrounded citation is harder to spot. The tool names a real case — one that exists in Westlaw or Lexis, has a proper citation, and appears in the right jurisdiction — but the case does not stand for the proposition the AI is using it to support. Because the citation is real, a quick cite-check may not reveal the problem; the attorney must actually read the cited authority and confirm it supports the specific claim. GPT-4 and even purpose-built legal tools can produce misgrounded citations at meaningful rates, making this the more dangerous failure mode in practice.
Misgrounding is especially common when a model retrieves a thematically related case and then overstates or subtly mischaracterizes its holding. The citation looks legitimate on a first pass, the case is real law, and the error only surfaces when someone reads the full opinion — which is exactly why these AI made-up citation problems are so consequential.
Why it matters more in law than in general search
Legal answers depend on binding, current, jurisdiction-specific authority. A plausible-sounding response that names the right legal concept but cites the wrong case can be pasted into a brief, quoted to a client in a memo, or relied on in contract drafting — and each of those uses carries professional-conduct exposure under the duty of competence and the duty of candor to the tribunal.
In ordinary web search a hallucinated result might cause inconvenience. In legal research, fabricated case law filed in federal court has already produced monetary sanctions, adverse rulings, and disciplinary referrals. The stakes make the distinction between factually wrong and merely misgrounded more than academic: both can end a motion, and misgrounded citations are the ones most likely to slip through a fast review.
| Failure type | Looks real? | Detectable without reading the case? |
|---|---|---|
| Factually incorrect statement | No citation given | Often yes |
| Fabricated case citation | No — case does not exist | Yes, on first cite-check |
| Misgrounded citation | Yes — case exists | No — requires reading the holding |
Key warning signs of AI hallucinated output:
- Citation format is correct but the case returns no results in Westlaw or Lexis
- The cited case exists but addresses a different legal question
- The holding is paraphrased more broadly than the actual opinion supports
- The statute cited was amended or repealed after the model’s training cutoff
How the hallucination rate is measured
The hallucination rate is measured by putting a fixed set of legal queries to each tool and having legal experts hand-score every answer as accurate, incomplete, or hallucinated. Stanford RegLab benchmarked hallucination rates in the Hallucination-Free? study, and that study tested 202 legal queries hand-scored by experts — making it the first systematic assessment of leading AI tools for real-world legal research tasks. The study was later peer-reviewed and published in the Journal of Empirical Legal Studies in 2025.
The Stanford RegLab benchmark
The Hallucination-Free? study was designed as a preregistered trial, meaning the researchers committed to their methodology before collecting data — an important safeguard against post-hoc adjustment. Each of the 202 queries was answered independently by each tool, and then a team of legal experts reviewed the responses without knowing which tool produced them. The blind scoring reduced the risk that familiarity with any product’s style would bias the evaluation.
Preregistration is what separates this benchmark from informal comparisons circulating in legal tech commentary. The study’s results were reproducible by design, and the full scoring rubric was disclosed, allowing vendors and critics to engage with specific methodological choices rather than disputing results in the abstract. Understanding the legal AI hallucination risk this benchmark documents is essential before deploying any of these tools in client matters.
Query types tested
Several hundred queries fell into four categories: general research and doctrinal questions, jurisdiction- or time-specific questions (such as circuit splits or recent statutory changes), false-premise questions that embedded a legal error the tool should catch, and factual-recall questions that asked for specific holdings or dates.
False-premise and jurisdiction-specific questions exposed the most hallucinations across every tool tested. When a query contains a built-in error, a well-calibrated tool should flag it; instead, several tools confirmed the false premise and built an answer on top of it. Jurisdiction-specific questions punished tools whose training data skewed toward federal law or majority jurisdictions.
What is scored as accurate, incomplete, or hallucinated
The three-way scoring system classified a response as accurate when it was both legally correct and well-grounded in the cited authority, incomplete when the tool refused to answer or gave an ungrounded or heavily hedged response, and hallucinated when the answer was factually incorrect or misgrounded.
The incomplete category matters because a tool can post a low raw hallucination number simply by declining to answer difficult queries. Refusal is a legitimate risk-reduction strategy from a product perspective, but it reduces a tool’s practical value for attorneys who need answers, not abstentions. The three-way split lets the benchmark reveal that trade-off explicitly rather than burying refusals inside a favorable accuracy figure.
How a query is scored step by step:
- The query is submitted to each tool under identical conditions with no supplemental prompting.
- Each response is stripped of tool branding before expert review.
- A legal expert reads the response and checks every cited authority against the primary source.
- The expert classifies the response as accurate, incomplete, or hallucinated using the preregistered rubric.
- A second reviewer independently scores a sample to establish inter-rater reliability.
- Disagreements are resolved by a third reviewer before results are tallied.
Legal AI hallucination rate by tool
The headline numbers from the Stanford RegLab benchmark are stark: Lexis+ AI hallucinated on about 17% of legal queries, Westlaw AI-Assisted Research hallucinated on about 33% of legal queries, and GPT-4 hallucinated on about 43% of legal queries in the same test set. Lexis+ AI answered accurately on 65% of queries — the highest accuracy of any tool tested — while Ask Practical Law AI gave incomplete answers on more than 60% of queries, the highest refusal rate in the study.
| Tool | Hallucination rate | Accuracy rate | Incomplete/refusal rate |
|---|---|---|---|
| Lexis+ AI | ~17% | ~65% | ~18% |
| Westlaw AI-Assisted Research | ~33% | ~42% | ~25% |
| Ask Practical Law AI | Low raw rate | Not reported | >60% |
| GPT-4 (no legal database) | ~43% | Low | Low |
| General chatbots (prior research) | 58–82% | Very low | Very low |
Lexis+ AI
In the Hallucination-Free? Stanford benchmark, Lexis+ AI posted the lowest hallucination rate of the purpose-built legal tools at approximately 17% and the highest accuracy rate at approximately 65%. That combination — low fabrication, high useful answers — is what distinguishes it from tools that manage hallucination by refusing to engage.
The roughly 18% of queries that fell into the incomplete category for Lexis+ AI means the tool still declines or hedges on a meaningful share of questions, but that share is far smaller than Ask Practical Law AI’s refusal rate. For practitioners doing doctrinal research across a broad range of query types, the Lexis+ AI profile represents the best measured balance in the study.
Westlaw AI-Assisted Research
Westlaw AI-Assisted Research hallucinated on about 33% of legal queries — roughly double the rate of Lexis+ AI on the same benchmark. Its accuracy rate was approximately 42%, meaning it answered correctly less than half the time. The gap relative to Lexis+ AI is large enough to influence tool selection, particularly for high-stakes work where every cited authority will be scrutinized.
The legal AI tool comparison across both platforms suggests that the difference is not merely a matter of model quality but also reflects how each product retrieves and summarizes authority. Westlaw’s longer average responses — discussed further in the next section — are part of the explanation for its higher misgrounding rate.
Ask Practical Law AI (Thomson Reuters)
Ask Practical Law AI presents a different risk profile: its raw hallucination count was relatively low, but it achieved that by giving incomplete or refused answers on more than 60% of queries — the highest refusal rate of any tool in the study. A tool that declines to answer most questions is conservative, but it is not reliably useful.
The practical implication for attorneys is that a low hallucination rate paired with a very high refusal rate does not mean the tool is safe in the way that a low hallucination rate paired with high accuracy would be. When Ask Practical Law AI does provide an answer, that answer may be reliable, but the majority of the time it will not give one at all, shifting the research burden back to the practitioner.
GPT-4 and general chatbots
GPT-4, tested without access to a legal case database, hallucinated on approximately 43% of legal queries in the Stanford benchmark. Earlier research on general-purpose chatbot accuracy found that general-purpose chatbots hallucinated on 58% to 82% of legal queries — a range that reflects how poorly models without retrieval augmentation handle jurisdiction-specific, binding-authority questions.
The comparison between GPT-4 at 43% and Lexis+ AI at 17% illustrates the practical value of retrieval-augmented generation: tying the model to a curated legal corpus reduces hallucination meaningfully, though it does not eliminate it. General chatbots remain the highest-risk option for any legal research task where the cited authority will be relied upon.
Why the tools differ
The differences in hallucination rates across legal AI tools come from a combination of retrieval-augmented generation quality, the underlying legal database, and how long and hedged each tool’s answers tend to be. Retrieval-augmented generation reduces but does not eliminate legal AI hallucinations, and longer AI answers correlate with higher hallucination rates — two findings that together explain much of the gap between Lexis+ AI and Westlaw AI-Assisted Research.
RAG reduces but does not eliminate hallucinations
Retrieval-augmented generation works by pulling relevant cases and statutes from a curated database before the language model drafts its answer, giving the model something real to summarize rather than relying entirely on training data. This is why all three purpose-built legal tools outperform raw GPT-4: the retrieval step constrains the answer space.
The problem is that retrieval does not guarantee accurate summarization. A tool can retrieve the correct case and then mischaracterize its holding in the generated summary — producing a misgrounded citation rather than a fabricated one, but a hallucination nonetheless. The Stanford benchmark showed this happening even in the lowest-hallucination tool, which means RAG is a necessary but not sufficient safeguard.
Answer length and verbosity
The study found that Westlaw AI-Assisted Research generated notably longer answers — approximately 350 words on average — compared to roughly 219 words for Lexis+ AI. Longer answers create more opportunities to misstate or misground authority, both because the model must make more factual claims and because extended prose tends to elaborate beyond what the retrieved sources directly support.
This correlation between answer length and hallucination rate is not unique to legal AI, but it has particular consequences in law: a verbose answer that discusses four cases confidently is harder to verify than a concise answer that cites one. Practitioners should treat unusually long AI responses as a signal to increase their verification effort, not as a sign of thoroughness.
Database and product design
The underlying case corpora available to LexisNexis and Thomson Reuters differ in coverage, update frequency, and the metadata used to rank retrieved passages. Prompt design — how the product frames the query to the underlying model — also shapes what gets retrieved and how confidently the model summarizes it.
Key factors that differentiate measured hallucination rates across tools:
- Depth and recency of the primary case and statute database
- How the retrieval step ranks and filters candidate passages
- Whether the product prompts the model to express uncertainty
- How aggressively the tool refuses queries outside its confident coverage
- The degree to which product testing included jurisdiction-specific and false-premise queries before release
Limits and criticism of the benchmarks
The benchmark numbers reflect specific product versions tested in mid-2024, and vendors have shipped updates since then, so the figures should be treated as directional rather than permanent. Ask Practical Law AI gave incomplete answers on more than 60% of queries in that snapshot — a result Thomson Reuters disputed partly on the grounds that the scoring did not fully credit the tool’s deliberate refusal policy as a feature rather than a failure.
The core methodological debate centers on how “hallucination” and “accuracy” are defined at the scoring stage. Thomson Reuters and other vendors questioned whether a conservative refusal should count against a tool as an incomplete response or be credited as responsible behavior. That definitional choice significantly affects the ranking: a tool that refuses 60% of queries can post a low hallucination rate while providing limited practical value, and different scoring conventions can flip which product looks best.
A related criticism concerns query selection. The 202 queries were drawn by researchers with particular expertise profiles, and critics argued that the mix of query types — especially the proportion of false-premise and jurisdiction-specific questions — may not reflect the distribution of tasks practitioners actually perform. If real-world use skews toward straightforward doctrinal research, the observed hallucination rates in the benchmark may overstate the risk practitioners face day to day.
Limits worth keeping in mind when reading the benchmark results:
- Tool versions change with product updates; a 2024 result may not reflect 2025 performance
- Inter-rater reliability was high but not perfect; close calls at the accurate/misgrounded boundary affect the final percentages
- The study was conducted in English on US law; performance on other jurisdictions or languages was not measured
- Academic benchmarks use open-ended queries; enterprise deployments often use constrained, templated prompts that may produce different rates
The fairest reading of the Stanford Hallucination-Free? study is that it established a rigorous, reproducible baseline and revealed meaningful differences between tools — not that it produced a permanent ranking. The relative ordering (Lexis+ AI outperforming Westlaw AI-Assisted Research, both outperforming general chatbots) is consistent with the underlying logic of each product’s design, which makes it credible as a directional finding even as the absolute percentages shift with model updates.
Real-world consequences: US court sanctions
US courts have sanctioned dozens of attorneys for filing AI-hallucinated citations, with tracked cases numbering in the hundreds and growing by several new cases per day. The pattern is consistent: a practitioner uses a chatbot or AI research tool, does not verify the output against primary sources, and files a brief citing cases that either do not exist or do not say what the brief claims.
the Lexis+ AI and Ask Practical Law AI systems produced incorrect information more than 17% of the time, while Westlaw’s AI-Assisted Research hallucinated more than 34% of the time
Stanford RegLab and HAI researchers
Sanctions are not hypothetical. Federal judges have imposed monetary penalties, required attorneys to submit sworn certifications that they verified AI-generated research, and in several instances referred lawyers to state bar disciplinary authorities. The court sanctions for AI citations tracker documents these cases across jurisdictions, and the pace of new filings has accelerated as AI tool adoption has grown.
ABA Model Rule 1.1 places the verification burden on the attorney, not the tool. The duty of competence requires lawyers to understand the technology they use and to check its output; the duty of candor to the tribunal requires that every cited authority be accurate and good law. Courts that have imposed sanctions have uniformly rejected the argument that relying on an AI tool shifts the professional responsibility to the vendor. Blaming the software has not shielded a single attorney from discipline in any reported case.
The professional exposure extends beyond brief-writing. A memo to a client that cites a hallucinated case and is later acted upon can give rise to malpractice claims. A contract that relies on a misstated statutory rule may not achieve the client’s intent. The hallucination rates documented in the Stanford benchmark — ranging from a fraction of queries for the best-measured tool to well over a third for general chatbots — translate directly into a material probability of professional error on any given research task that is not independently verified.
How to lower your hallucination risk
The hallucination risk is lowered by verifying every cited authority in a primary-source database, preferring tools with lower measured rates, and treating AI output as a draft that requires attorney review before any use. Retrieval-augmented generation reduces but does not eliminate legal AI hallucinations, which means even Lexis+ AI — hallucinating on about 17% of legal queries — requires independent verification of every citation it produces.
Verify every citation against primary sources
Every case or statute an AI tool cites should be pulled directly in Westlaw, Lexis, or a comparable primary-source database before it appears in any filing, memo, or client advice. The verification workflow has three distinct steps: confirm the citation format resolves to a real document, confirm the document is still good law (not overruled, superseded, or amended), and confirm the holding or language actually supports the specific proposition for which it is being cited.
The third step is the one most often skipped. It is not enough that a case exists and is good law; misgrounding — the more dangerous failure mode — only surfaces when someone reads the relevant section of the opinion or statute and tests whether it genuinely supports the claim. This takes more time than a quick Shepard’s or KeyCite check, but it is the only step that catches the misgrounded-citation failure mode the benchmark identified as hardest to detect.
Choose tools with lower measured rates
When selecting a legal AI research tool, weigh hallucination rate together with accuracy rate and refusal rate as a three-variable profile rather than relying on any single number. A tool with a low hallucination rate that achieves it by refusing most queries is not the same as a tool with a low hallucination rate and high accuracy.
The Stanford benchmark data suggests that Lexis+ AI’s combination of approximately 17% hallucination and approximately 65% accuracy makes it the most useful profile of the tools tested — low fabrication and high useful output together. If your practice involves heavy use of jurisdiction-specific or false-premise-sensitive queries, the gap between tools widens further, making tool selection a more consequential decision than in general doctrinal research.
Treat AI as a draft, not authority
No current legal AI tool is hallucination-free; the best-measured one still fabricates or misgrounds on roughly one query in six. That rate is low enough to make AI genuinely useful for surfacing cases, identifying legal frameworks, and drafting initial analysis — but high enough to make it professionally dangerous to treat AI output as finished work product.
The practical posture is to use AI to accelerate the research process and generate drafts, then apply the same independent verification that would be required for any associate’s first draft: every citation confirmed, every holding read, every statutory reference checked against the current codified version. The professional responsibility belongs to the attorney who signs the filing, regardless of which tool helped produce the first draft.