الفصل 25/29
How to Evaluate a Legal AI Vendor (Red Flags)
A structured red-flag inventory across four risk surfaces: security, legal/compliance, model/output, and commercial. Walk every vendor through these.
- وقت القراءة: 14 min
ما يغطيه هذا الفصل
- Security & Confidentiality
- Legal & Compliance
- Model & Output
- Sales & Commercial
فصول الدورة مكتوبة بالإنجليزية. أما بقية الأكاديمية فمترجمة.
TL;DR
Most legal AI procurement happens on vibes. Below is a structured red-flag inventory across four risk surfaces: security, legal/compliance, model/output, and commercial. Walk every vendor through these. The ones who flinch are the ones to avoid.
Security & Confidentiality
- No SOC 2, ISO 27001, or equivalent audit reports.
- Vague answers on data residency.
- Customer data used to train the vendor's models.
- No tenant-level isolation between firms.
- Unclear breach notification obligations.
- No deletion guarantees on offboarding.
Legal & Compliance
- Refusal to sign a DPA.
- No clear position on bar-rule compliance per jurisdiction.
- Privilege not preserved through their pipeline.
- Disclaimer that disowns all output - while marketing it as 'lawyer-grade.'
- No audit trail of what the AI did, when, and on which sources.
Model & Output
- Won't tell you which underlying model they use.
- No mechanism to flag uncertainty in outputs.
- No source citations in answers.
- Citations that look real but don't resolve.
- Same prompt yields wildly different answers run-to-run.
- No way to constrain to your jurisdiction.
Sales & Commercial
- Demos on perfectly curated test cases - never your own files.
- Pricing only available 'on call.'
- Multi-year lockups with no exit ramp.
- Customer references are all anonymized.
- ROI claims with no methodology.
- Refusal to support a 30-day pilot.
For the full 45-item checklist with scoring rubric, see the companion blog post: 45 Red Flags for Legal AI Vendor Evaluation.