Ledger
Detect quality regressions in AI systems before they reach users.Our evaluation platform. Ledger runs regression suites against every prompt, model or code change and blocks releases that make quality worse.
- running 2,000 casesbaseline main@a91f3
- ✓ invoice.total_amount99.4% (+0.2)
- ✓ invoice.line_items97.1% (+1.8)
- ✓ delivery_slip.consignee98.8% (±0.0)
- ✗ customs.hs_code · source=fax91.2% (−3.4)
- grader agreement0.93 vs human labels
- gate failed1 slice below threshold · 14 cases changed
- grader agreement with human reviewers
- 0.91
- grader agreement with human reviewers
- median full-suite run in CI
- ~9 min
- median full-suite run in CI
- of our releases gated by Ledger
- 100%
- of our releases gated by Ledger
- cases in our largest client suite
- 2,000
- cases in our largest client suite
Overview
Why we built Ledger.
Changing a prompt, a model or a retrieval setting can quietly break things that used to work. Ledger turns quality into something you can measure. Test sets are built from real production cases, scored with a mix of deterministic checks, model-based graders and human review, and tracked over time.
Ledger plugs into CI. A pull request that lowers accuracy on faxed invoices, or makes a support agent break policy once in 1,600 replayed inquiries, fails the build in the same way as a failing unit test.
Capabilities
Core capabilities.
Datasets from production
Promote real cases and corrections into versioned test sets in one click.
Mixed grading
Exact checks, schema validation, model-based graders and human review queues in one suite.
CI gates
Per-metric thresholds that fail pull requests on regressions, with a diff of the cases that changed.
Slice analysis
Break results down by document type, source channel, language or any metadata field.
Grader calibration
Model graders are checked against human labels so you know when a grader can be trusted.
Production monitoring
Sampled live traffic scored continuously, with alerts when quality drifts.
Architecture
System architecture.
A simplified view of the principal layers and components.
Built with
Sources
Production tracesLabelled setsHuman correctionsSynthetic casesSuites
Versioned datasetsTest definitionsThresholdsSlicesGraders
Deterministic checksModel gradersHuman reviewCalibrationOutputs
CI statusDiff reportsDashboardsDrift alerts
In production
Client projects built on Ledger.
Contact
Have an AI product in mind?
Share what you would like to build, and we will help turn the idea into a production-ready product.
No technical specification required.
What happens next
- 1
Share your idea
Describe the problem, the users and the outcome you need.
- 2
Discovery call
We review your requirements together and clarify scope and constraints.
- 3
Proposal
You receive a clear scope, technical approach and delivery plan.