Legal Benchmarks
How we build and score a legal benchmark, and the public legal benchmarks that came before it, from 2021 to 2026.
Benchmark Framework
How we build and run a legal benchmark: the task prompts, the scoring rubric, the public frameworks it draws on, how averages are computed, and where it stops.
Scoring Model Routing and Jev
Two parts sit between a prompt and an answer: the router that decides which model runs a task, and Jev, a calibrated judgment service we test as a checker. Neither is trusted on faith. Each is scored on the same tasks and the same rubric, and no score from this step is shown here yet.
- 1
Route Against a Fixed Model
Each task runs through the router and, separately, through each fixed model under the same prompt. Routing has to beat a single model, not an empty baseline.
- 2
Quality per Dollar
A route is scored on rubric quality divided by cost per task, counting every billable token. It is the same index as the Performance chart on the Labs page.
- 3
Boundary Tests for Jev
Jev is tested where a checker can be fooled, such as a fabricated citation next to an honest one. The cases it gets wrong are reported alongside the cases it gets right.
- 4
Its Own Scale
Routing and Jev results are reported on their own scale and never folded into the task benchmark score. Model versions and the run date are named.
Two Evaluations, Two Scales
The Task Benchmark
11 legal task categories. On the work a lawyer actually hands over, who produces the better output?
The Cross-Domain Study
25 benchmarks in 5 domains: legal, tax, journalism, general capability, and safety and values. Against the frontier models on their own ground, where does a legal-specialised engine stand?
What the Rubric Covers
- Sharia
- Statute
- Forum
- Clause Construction
- Risk
- Hallucination
- Formatting
- Brevity
- Partner-Readiness
- Source Linking
Partner-Readiness
An answer that is correct but needs an hour of restructuring before a partner would sign it has not done the job.
Source Linking
A citation that does not resolve is worse than no citation, because it transfers the checking cost to the reader without telling them.
Public Frameworks the Study Draws On
Stanford LegalBench opens in a new tab
Legal reasoning across a broad task taxonomy
All-pass rubric grading of long-horizon legal work
VLAIR (Vals AI) opens in a new tab
Independent industry evaluation of legal AI systems
Arabic legal reasoning, which most legal benchmarks do not cover
BigLaw Bench opens in a new tab
Tasks drawn from large-firm practice
Contract clause extraction and understanding
LegalCiteBench opens in a new tab
Citation accuracy and resolvability
LawBench / LexEval opens in a new tab
Multilingual legal capability
Legal Benchmarks opens in a new tab
Cross-vendor methodology
What This Benchmark Does Not Do
It Is Not Independent
HAQQ builds it, runs it, and competes in it. That is why the methodology is published and the per-task results are visible rather than summarised.
It Does Not Cover Every Practice Area
The task benchmark is weighted toward corporate and commercial drafting, research and client-facing explanation. Other areas are covered thinly or not at all.
It Does Not Cover Every System
Several legal AI vendors publish no evaluations and offer no self-serve access. A system absent from a table was not beaten by us; it was not run.
It Is a Point in Time
Every entrant ships new versions. A score is the version named at the run date and nothing more.
It Measures Output, Not Deployment
Security posture, data residency, support and integration depth decide whether a system can be used at all, and none of them appear in the benchmark.
Historical Data
The legal benchmarks that have shaped how AI models are measured, from the first public datasets in 2021 to 2026.
Scores are not shown. Each benchmark uses its own tasks and metrics, so scores from different benchmarks cannot be compared. Every entry links to its primary source.
2021
2023
2024
2025
2026
