🔍 Read the full analysis: Five Things To Know About Z.ai’s Legal AI Benchmark Rank And Pass Rate on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria pass rate on Vals AI’s Harvey Legal Agent Benchmark leaderboard, ranking seventh among 81 models. It passed 92.5% of individual criteria, but a task counts as a pass only when every required criterion is met. Z.ai’s claimed 12.5-percentage-point gain over its base model matches the listed scores; its broader “state-of-the-art among open-source models” claim is not supported by a stated comparison set in the cited material.
Z.ai’s GLM-5.3-FlashX-Legal achieved a 19.17% all-criteria pass rate on Vals AI’s archived leaderboard for Harvey’s Legal Agent Benchmark, placing it seventh among 81 models. The result means it met every rubric requirement on about one in five benchmark tasks—not that it performed at 19% on every measure: Vals separately reports a 92.5% pass rate across individual criteria.
The leaderboard applies an all-or-nothing task rule. Harvey says a task passes only when the model meets every rubric criterion for the requested work product. A missed deadline, citation, recommendation or other requirement can make the entire task a failure, even if the remaining work is correct. The 19.17% figure is the average task pass rate from two AI judges grading a private holdout set of 120 tasks.
On the same archived Vals leaderboard, GLM Legal scored 92.5% on individual criteria. These measures answer different questions: the criterion rate records performance on separate rubric items, while the task rate requires all of those items to pass together. The gap between them is material for anyone interpreting the headline score as a measure of usable legal work.
The leaderboard lists GLM Legal at $0.15 per test, compared with $0.57 for the listed base model, GLM 5.3 Flash. That is a benchmark testing cost, not Z.ai’s API price. Z.ai’s pricing page separately lists $0.37 per million input tokens and $1.25 per million output tokens. The company says the legal model is built on GLM-5.3-Flash and was adapted for legal workflows.
A Stronger Score, With Limits
The result gives law firms and legal technology buyers a concrete, third-party benchmark entry to compare with other systems, but the task pass rate remains below one in five. Because the test uses an all-pass rule, that figure does not mean the other tasks were wholly incorrect; it does indicate that each failed task had at least one unmet requirement. Conversely, the 92.5% criterion rate does not establish that a work product is safe to use: a single consequential error can matter more than many correctly handled details.
Vals lists 19.17% for GLM Legal and 6.67% for its base model, a difference of 12.50 percentage points. That supports Z.ai’s stated improvement on this leaderboard version. Z.ai also calls the result “state-of-the-art performance among open-source models,” but the source material does not identify the comparison set, and it does not establish that the legal model’s weights are publicly available. That claim should not be treated as a verified ranking across open-source systems.
For procurement, benchmark performance is only one part of the decision. Firms would still need to check citation accuracy, confidentiality and data handling, integration requirements, and human review processes. The benchmark is evidence about performance on a defined set of tasks, not a guarantee that the model can independently provide legal advice or produce work ready for clients.
Products Worth Considering
Trezor Safe 7 Crypto Hardware Wallet
LG UltraGear 27G640A-B 27-Inch QHD IPS Gaming Monitor, 300Hz, 1ms, G-SYNC Compatible, FreeSync Premium, DisplayHDR 400
Logitech MX Keypad Programmable Macro Pad, Graphite
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Tasks
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. Its public set contains 1,250 tasks across 24 practice areas, with more than 75,000 expert-written rubric criteria. Each task pairs instructions with client-matter materials and asks an agent to produce a work product. Harvey describes the criteria as binary checks covering matters such as facts, conclusions, citations, deadlines, dollar amounts and formatting.
For public leaderboard evaluations, Vals AI uses a private 120-task holdout designed to reflect the public mix. Internet access is disabled, and two AI judges grade submissions; the reported task score averages their pass rates. Results therefore reflect a specific benchmark setup and grading process, rather than a direct test of every legal task a firm might encounter.
Z.ai’s developer documentation describes GLM-5.3-FlashX-Legal as optimized for document understanding, legal reasoning and task execution, including research, contract review and drafting initial drafts. Its October 13, 2026, AutoClaw blog post announced an application-based beta. The documentation says the model underwent legal-domain training and recommends using it as an agent connected to services providing legal and company information.
“A task is marked complete only if every criterion passes.”
— Harvey, describing its benchmark grading rule
What the Benchmark Cannot Establish
The benchmark does not establish how GLM Legal performs across the full range of real legal matters, jurisdictions or firm workflows. The cited results come from a 120-task private holdout under specified grading conditions. Vals uses two AI judges, and the material supplied does not provide enough detail to determine how often those judges disagreed or how the result would change under human-only review.
The cited sources also do not specify the model’s full data-handling terms, where client information would be processed, or what contractual protections apply to law firms. Z.ai’s documentation describes draft-oriented uses, but the supplied material does not establish that outputs are independently verified or suitable for filing or client delivery without review. The source author also reports a four-task spot check in which the model cited rulings the author could not find; that small, non-benchmark sample is an observation, not a general performance rate.
Finally, Z.ai’s “state-of-the-art among open-source models” description cannot be fully checked from the supplied evidence: no comparison set is named, and no published weights for the legal model are identified. The archived leaderboard establishes a rank for that version and date, not a permanent standing.
Questions Before Firm Adoption
The immediate next step for prospective users is to review Z.ai’s beta and deployment terms, then test the system on representative work under controlled conditions. Firms evaluating it will need clear answers on client-data processing, retention and access, as well as the legal sources and internal systems connected through the recommended agent setup.
Further leaderboard updates could change the model’s position as Vals adds results or revises its benchmark. Buyers should compare versions and scoring rules before treating rankings as directly comparable. Any use in legal work also calls for qualified professional review, particularly for citations, deadlines, conclusions and other details where an error could affect a client.
For now, the confirmed result is narrower: GLM Legal placed seventh on the archived leaderboard, with a 19.17% all-criteria task rate and a 92.5% individual-criteria rate. Whether those scores translate into dependable, cost-effective performance in a particular firm’s workflow remains to be tested.
Key Questions
What does GLM Legal’s 19.17% score mean?
It means the model met every rubric criterion on 19.17% of the benchmark’s tasks, averaged across two AI judges. A task with even one failed criterion counts as a task failure.
How can the criteria pass rate be 92.5%?
The 92.5% figure measures individual rubric criteria, while 19.17% measures whole tasks that passed every criterion. A system can pass most separate checks but still miss at least one requirement on many tasks.
Did Z.ai’s model improve on its base model?
On Vals AI’s archived leaderboard, GLM Legal scored 19.17% and GLM 5.3 Flash scored 6.67%, a difference of 12.50 percentage points. This supports the stated improvement for that benchmark version; it does not establish a gain on every legal task.
Does the ranking prove the model is ready for legal work?
No. The leaderboard measures performance on a defined set of tasks. It does not establish reliability for every matter or replace checks on citations, client-data handling and review by qualified legal professionals.
Source: ThorstenMeyerAI.com
