All projects
AI evaluationIn productionSince 2025

Ledger

Detect quality regressions in AI systems before they reach users.Our evaluation platform. Ledger runs regression suites against every prompt, model or code change and blocks releases that make quality worse.

ledger · suite docs.logistics · PR #412Example trace
  1. running 2,000 casesbaseline main@a91f3
  2. ✓ invoice.total_amount99.4% (+0.2)
  3. ✓ invoice.line_items97.1% (+1.8)
  4. ✓ delivery_slip.consignee98.8% (±0.0)
  5. ✗ customs.hs_code · source=fax91.2% (−3.4)
  6. grader agreement0.93 vs human labels
  7. gate failed1 slice below threshold · 14 cases changed
grader agreement with human reviewers
0.91
grader agreement with human reviewers
median full-suite run in CI
~9 min
median full-suite run in CI
of our releases gated by Ledger
100%
of our releases gated by Ledger
cases in our largest client suite
2,000
cases in our largest client suite

Overview

Why we built Ledger.

Changing a prompt, a model or a retrieval setting can quietly break things that used to work. Ledger turns quality into something you can measure. Test sets are built from real production cases, scored with a mix of deterministic checks, model-based graders and human review, and tracked over time.

Ledger plugs into CI. A pull request that lowers accuracy on faxed invoices, or makes a support agent break policy once in 1,600 replayed inquiries, fails the build in the same way as a failing unit test.

Capabilities

Core capabilities.

01

Datasets from production

Promote real cases and corrections into versioned test sets in one click.

02

Mixed grading

Exact checks, schema validation, model-based graders and human review queues in one suite.

03

CI gates

Per-metric thresholds that fail pull requests on regressions, with a diff of the cases that changed.

04

Slice analysis

Break results down by document type, source channel, language or any metadata field.

05

Grader calibration

Model graders are checked against human labels so you know when a grader can be trusted.

06

Production monitoring

Sampled live traffic scored continuously, with alerts when quality drifts.

Architecture

System architecture.

A simplified view of the principal layers and components.

Built with

PythonFastAPIPostgreSQLDuckDBReactGitHub Actions
  1. Sources

    Production tracesLabelled setsHuman correctionsSynthetic cases
  2. Suites

    Versioned datasetsTest definitionsThresholdsSlices
  3. Graders

    Deterministic checksModel gradersHuman reviewCalibration
  4. Outputs

    CI statusDiff reportsDashboardsDrift alerts

In production

Client projects built on Ledger.

Contact

Have an AI product in mind?

Share what you would like to build, and we will help turn the idea into a production-ready product.

No technical specification required.

What happens next

  1. 1

    Share your idea

    Describe the problem, the users and the outcome you need.

  2. 2

    Discovery call

    We review your requirements together and clarify scope and constraints.

  3. 3

    Proposal

    You receive a clear scope, technical approach and delivery plan.