Skip to content
HAQQ
Kapitel 25/30

How to Evaluate a Legal AI Vendor (Red Flags)

A structured red-flag inventory across four risk surfaces: security, legal/compliance, model/output, and commercial. Walk every vendor through these.

CHAPTER 25of 30How to Evaluate a Legal AIVendor (Red Flags)
A Victorian patent-medicine trade card advertising Carter's Liver Bitters with a celebrity endorsement.
An endorsement, a promise nobody can test, and no way to check what is in it. The red flags in this chapter are the same red flags. The pitch deck has better typography. Boston Public Library · Public domain · via Wikimedia Commons

Was dieses Kapitel behandelt

  1. Security & Confidentiality
  2. Legal & Compliance
  3. Model & Output
  4. Sales & Commercial

Die Kurskapitel sind auf Englisch verfasst. Der Rest der Academy ist übersetzt.

TL;DR

Most legal AI procurement happens on vibes. Before you compare anyone, know where they sit in The Legal AI Vendor Landscape. Below is a structured red-flag inventory across four risk surfaces: security, legal/compliance, model/output, and commercial. Walk every vendor through these, then score the survivors against real work in Practitioner-Led Benchmarks, not their own demo. The ones who flinch are the ones to avoid.

Security & Confidentiality

  • No SOC 2, ISO 27001, or equivalent audit reports.
  • Vague answers on data residency - run their answers through the GDPR Compliance Checklist.
  • Customer data used to train the vendor's models.
  • No tenant-level isolation between firms.
  • Unclear breach notification obligations.
  • No deletion guarantees on offboarding.

Legal & Compliance

  • Refusal to sign a DPA.
  • No clear position on bar-rule compliance per jurisdiction.
  • Privilege not preserved through their pipeline.
  • Disclaimer that disowns all output - while marketing it as 'lawyer-grade.'
  • No audit trail of what the AI did, when, and on which sources.

Model & Output

  • Won't tell you which underlying model they use.
  • No mechanism to flag uncertainty in outputs.
  • No source citations in answers.
  • Citations that look real but don't resolve.
  • Same prompt yields wildly different answers run-to-run.
  • No way to constrain to your jurisdiction.

Sales & Commercial

  • Demos on perfectly curated test cases - never your own files.
  • Pricing only available 'on call,' instead of published like Legal AI Pricing in 2026.
  • Multi-year lockups with no exit ramp.
  • Customer references are all anonymized.
  • ROI claims with no methodology.
  • Refusal to support a 30-day pilot.

For the full 45-item checklist with scoring rubric, see the companion blog post: 45 Red Flags for Legal AI Vendor Evaluation.

Chapter Quiz

Test your understanding of the concepts covered in this chapter.

Question 1 of 6

The chapter says most legal AI procurement happens on vibes. What does it put in place of that?