How Sentinel gets sharper on autopilot

Sentinel is a pipeline, not a score

It's easy to call Sentinel "our quality scorer," but that undersells it. For every image it plans what to check, runs ~20 vision/ML checks, has LLM judges grade each dimension, and a final reviewer returns a verdict — pass / soft-fail / hard-fail — plus a 0–100% weighted pass rate (pass = 100%, soft-fail = 70%, hard-fail = 0%, with hard gates called out separately). That percentage is the "60% / 95%" you'll see below — it's a pass rate, not model confidence.

Runflow — the fuel

Everything starts with real work. Every Runflow job — headshots, product shots, background removal, virtual try-on — produces a real output, in the exact mix of niches our customers actually run. For each output image, Sentinel writes an evaluation. On top of that we capture two human signals: a 👍/👎 on the run, and a separate 👍/👎 on whether Sentinel itself got it right.

What turns raw outputs into a learning signal is the disagreements:

This is production data, not lab data — the real distribution, with real human reactions attached. Today we learn only from our own internal and consented traffic; customer opt-in is on the roadmap, behind a clear legal basis. That discipline is a feature — the moat only compounds if it's built on data we're actually allowed to use.

Arena — judge the judges

The trick is subtle. I can't cheaply ask a human "is this image good?" at scale and get consistent truth. But I can ask "which of these evaluators do you most agree with?" — and that tells me which judge sees quality the way we want.

So Arena takes one real output and has a panel score it in one shared format. A human then picks the evaluator whose verdict they most agree with, and a leaderboard ranks the judges by how often humans agree.

Benchmark — the frozen scoreboard

Two things share the word "benchmark," so let's split them:

The Benchmark is live today as a command-line tool: ~64 hand-picked cases across 8 niches, frozen so they never change between runs. Run a candidate Sentinel campaign/config against it and it tells you, per niche, whether that candidate improved, held, or broke something — and it flags pass→fail regressions. It even labels each call:

The rule that keeps it honest: we never train on this set. It's a measuring stick, not a textbook. Still building: exact Sentinel build/version stamping · an always-on dashboard · a frontier panel inside the benchmark · a release gate that blocks regressions.

The autopilot — an autonomous researcher

This is the part we never talked about, and it's the reason the title says "autopilot." Once the Benchmark is a trustworthy scoreboard, the next step is to point a Claude Code agent at it and let it propose changes to Sentinel — using the knobs Sentinel already exposes (judge prompts, which model runs each stage, how hard each stage "thinks," the analysis plan) toward one objective: higher agreement with our human-gold benchmark, and lower latency.

No auto-merge. No weight training. It never ships itself to production.

The data moat — the payoff

That trustworthy scoreboard runs on human-adjudicated gold — and that's the moat. The scarce, hard-to-copy asset is human-adjudicated gold on our own internal/consented production distribution, by niche — and it deepens with every case we can legally keep.

Defensibility = how many niches we cover × how good our labels are × the governance to use the data responsibly. That consent discipline isn't a constraint around the moat — it's part of it. And it's the same gold that lets the autopilot run safely: more gold → a truer scoreboard → safer autonomy → a sharper Sentinel.

The loop, and what keeps it honest

Put it together and it's a flywheel: a sharper Sentinel makes Runflow's scoring and Arena's comparisons cleaner → better cases and cleaner curation → a bigger benchmark → the autopilot sharpens Sentinel again.