How we build and run a legal benchmark: the task prompts, the scoring rubric, the public frameworks it draws on, how averages are computed, and where it stops.
1
Same Prompt, Same Task
Each system is given the same prompt for a task a lawyer would recognise: draft this NDA, research this question, explain this provision to a client.
2
A Fifty-Point Rubric
The output is scored against a fifty-point rubric. Partner-readiness and source linking are the two criteria that separate a legal evaluation from a general one.
Partner-ReadinessSource Linking
3
Public Frameworks
The cross-domain study draws on public frameworks rather than instruments we wrote ourselves, so the legal columns can be checked by someone outside HAQQ.
Stanford LegalBenchCUADVLAIR
4
All-Pass Grading
A task counts only when every criterion in its rubric is met. A near-miss scores zero, not partial credit, because output that is ninety per cent correct still has to be checked line by line.
5
How Averages Work
A domain average is the mean of that domain's benchmarks. The overall average is the mean of twenty-four benchmarks, leaving out adversarial testing. A blank cell is excluded from every average and is never treated as a zero.
6
Report and Analysis
Rows where HAQQ is not first are shown exactly like rows where it is: same weight, same position, no reordering and no omission. Where a figure is derived, the derivation is stated.
Task Benchmark · Out of 50Cross-Domain Study · Out of 100
Illustration only. Bar lengths carry no data.
The two scales are not interchangeable. A score out of 50 and a score out of 100 measure different work on different rubrics, and must never be read as comparable.
Scoring Model Routing and Jev
Two parts sit between a prompt and an answer: the router that decides which model runs a task, and Jev, a calibrated judgment service we test as a checker. Neither is trusted on faith. Each is scored on the same tasks and the same rubric, and no score from this step is shown here yet.
1
Route Against a Fixed Model
Each task runs through the router and, separately, through each fixed model under the same prompt. Routing has to beat a single model, not an empty baseline.
2
Quality per Dollar
A route is scored on rubric quality divided by cost per task, counting every billable token. It is the same index as the Performance chart on the Labs page.
3
Boundary Tests for Jev
Jev is tested where a checker can be fooled, such as a fabricated citation next to an honest one. The cases it gets wrong are reported alongside the cases it gets right.
4
Its Own Scale
Routing and Jev results are reported on their own scale and never folded into the task benchmark score. Model versions and the run date are named.
Historical Data
The legal benchmarks that have shaped how AI models are measured, from the first public datasets in 2021 to 2026.
Scores are not shown. Each benchmark uses its own tasks and metrics, so scores from different benchmarks cannot be compared. Every entry links to its primary source.
2021
3 benchmarks
CUAD
Launched 2021
Contract review: highlighting the clauses in a contract that matter to a human reviewer, with over 13,000 expert annotations.
Arabic legal argument reasoning: over 13,000 commercial court cases from Saudi Arabia, with the facts, the court's reasoning, the verdict and the cited clauses.
An agentic benchmark for US legal research: agents search case law, the web and documents, and are graded on expert-written rubrics across eight practice areas.
