🔍 Read the full analysis: The Law Model’s 8/100 Score In A Test Of 42 AI Setups on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A test run by Lexassay, a legal AI assessment project, gave GLM-5.3 FlashX Legal 8 out of 100 across 15 legal tasks and said it fully solved none. The test’s operator disclosed that his organization runs Lexassay and that Claude agents graded answers; its results are a spot check, not a definitive ranking of legal AI systems.
GLM-5.3 FlashX Legal scored 8 out of 100 in a test of 42 AI setups answering 15 legal questions, according to results published November 4 by Lexassay, a testing project run by the article’s author. The model, which its maker Z.ai describes as optimized for legal workflows, fully solved none of the tasks and gave no answer on two, raising questions about how much users should rely on legal-specific chatbots for citations, legal analysis or deadlines.
Lexassay said the test included 10 published questions and five hidden questions, covering German, U.S. and EU law. Subjects included calculating deadlines, changes to laws, court decisions and questions that contained false assumptions. The models answered without internet access or tools, meaning the test measured responses from the models themselves rather than research aided by a search system.
The project reported that GLM-5.3 FlashX Legal received zero on the hidden tasks. Its citation review found 21 real citations, 25 that were misattributed and one invented citation. Lexassay said the model used its full allowance of 64,000 tokens on two German-law tasks but returned no answer. Its base model scored 3, while the legal version scored 8.
Lexassay’s reported top scores were Claude Opus 5.5 at 79, GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Even the top model met every criterion on only eight of the 10 public tasks, according to the results. Across all published runs, the test said it verified 2,644 citations as real and identified 804 as invented or as not supporting the claim made. These counts describe the test’s reviewed citations; they are not a rate of error for all legal AI use.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Errors Can Affect Legal Deadlines
The test highlights a specific risk in legal AI: a response can sound confident while getting a date, legal authority or the meaning of a ruling wrong. In some legal processes, a missed filing deadline can have serious consequences. Lexassay gives an example involving two German civil appeals: a public holiday would move the Munich deadline by a day, while the deadline in Berlin would remain unchanged. That example illustrates why legal dates can depend on jurisdiction and local rules, but it is not a substitute for checking an actual case with a lawyer.
The citation results also matter because references are often how readers check legal claims. A ruling may exist but fail to support the chatbot’s description of it; a citation can also be invented. The test’s findings do not establish how frequently these errors occur in everyday use, but they show that a legal label or polished answer alone is not proof of reliability.
For readers, the results support treating chatbot output as a starting point rather than legal advice. A deadline, claim value or legal position should be checked against authoritative sources and, where consequences are material, with a qualified lawyer. Legal questions are jurisdiction-specific, and the test itself covered only a limited set of tasks.
Products Worth Considering
Apple MacBook Pro 16.2-inch with M5 Pro, 18-Core CPU, 20-Core GPU, 48GB Memory, 1TB SSD, Wi-Fi 7, Silver
Logitech MX Master 4 for Mac Wireless Ergonomic Mouse, Space Black
Logitech MX Anywhere 3S for Mac Compact Wireless Mouse, Space Gray
As an affiliate, we earn on qualifying purchases.
How Lexassay Scored the Models
Lexassay is run by Thorsten Meyer AI, which disclosed that it operates the test. The article also said several Claude agents graded answers blind, and that a Claude model ranked first. Those relationships are relevant when interpreting the results. Lexassay said its method page publishes the questions, gold answers and citation checks so readers can review the grading.
Each answer was compared with a gold answer using three to six weighted criteria. Citations were checked against official databases and categorized as real, misattributed, invented or not findable. According to the method description, each invented or misattributed citation reduced a task’s points by a quarter, and an answer without a legal-advice disclaimer or referral to a lawyer lost 5 percent. Different reasoning-effort settings were tested separately, so a model could appear more than once among the 42 setups.
The publisher describes Lexassay as a spot check, not a strict benchmark: it tested one answer per model and task and cautioned that differences of a few points may be noise. The article also noted a separate Vals AI legal-agent benchmark, which supplies documents to models and ranked Mistral Large 4 eighth, ahead of Claude Opus 5.5 and GPT-6 Astra. The differing results reflect different test designs; they do not establish that one ranking is universally correct.
“Optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.”
— Z.ai, as quoted in the Lexassay report
Limits of the Reported Scores
The results do not establish how the models would perform with access to legal databases, web search, supplied case documents or professional review. Lexassay tested models without internet access or tools, while other legal AI products may combine models with retrieval systems and human workflows. The report also provides only one answer for each model-task pairing, so individual outputs may vary.
The project’s disclosed relationship with its operator and its use of Claude agents to grade answers are factors readers may weigh. Blind grading and published answer materials may help scrutiny, but the source does not describe an independent replication of the results. The report also does not establish a general error rate for legal AI or show that the scores predict performance on real client matters.
Some details in the supplied report are incomplete: its discussion of GLM-5.3 FlashX Legal’s standing among other models ends mid-sentence. The report gives Mistral Large 4’s score as 10 and describes citation problems, but does not provide its complete score breakdown in the supplied material. The number and precise identities of all 42 setups are also not listed there.
Review the Methods and Answers
Lexassay says its method page makes the public questions, gold answers and citation checks available for review; the five hidden tasks have scores published without the answers. Readers seeking to assess the findings can compare the published responses with the stated criteria and official legal sources, while keeping in mind that the project itself says the test is a spot check.
The report does not announce a date for another test or an independent audit. Further clarity would come from replication across more prompts, repeated runs and tool-enabled legal systems, as well as publication of full results for each setup. Until then, the reported scores describe performance on this specific set of questions and conditions, not a guarantee about how any model will handle a particular legal matter.
Key Questions
What did GLM-5.3 FlashX Legal score?
Lexassay reported a score of 8 out of 100 across its test, with zero on the five hidden tasks. It said the model fully solved none of the 15 questions and returned no answer on two.
What was the test designed to measure?
It tested responses to 15 legal questions covering German, U.S. and EU law, including deadlines, legal changes, court decisions and false premises. Models answered without internet access or tools.
Does the score prove that legal AI systems are unreliable in every setting?
No. Lexassay describes the work as a spot check based on one answer per model and task. It does not measure every legal product, tool-enabled workflow or real-world use.
Who ran and graded the test?
Thorsten Meyer AI runs Lexassay, a relationship disclosed in the report. The publisher said several Claude agents graded answers blind and that a Claude model ranked first; readers can review the published method and answers for the public tasks.
Can someone rely on a chatbot for a legal deadline?
The report advises against taking a deadline from a chatbot. Deadlines can depend on jurisdiction and case details; anyone facing a filing deadline or other legal consequence should verify it with authoritative sources and consult a lawyer.
Source: ThorstenMeyerAI.com
