How a Run Works

Sample, replay, judge, aggregate — and what each phase leaves behind.

A Run is one comparison on one Monitor. It goes through the same phases every time, and every phase leaves a receipt on the Run report. Nothing in it depends on the order the judges answered in, and nothing is resized to fit: a Run that cannot fund the requested sample fails rather than quietly shrinking and publishing a full-sized savings figure.

1. Sample

PeerLM draws a sample of real prompts from the Monitor's traffic — 150 by default, never fewer than 30. Prompts that cannot be replayed faithfully (tool calls, images, truncated spans) are excluded and the exclusion is counted, so the coverage figure on the report is the share of your traffic the verdict actually rests on.

2. Replay

Each candidate model answers every sampled prompt. In observed mode — the default — the incumbent is not called at all: each challenger is compared one-to-one against the exact response your production model really gave. In replayed mode the incumbent is regenerated too, which is what Prompt CI uses so both prompt variants run on the same model under the same conditions.

3. Judge

A panel of five judges, rotated per item from a pool of eight frozen on the Monitor, sees each pair and returns a winner, a failure tag, and one sentence. Judging is pairwise only — there are no 1–10 scores — and PeerLM selects the judges: a model from the vendor under test never judges it. Any objective validators you configured (exact text, JSON paths, a schema, a safe regex) run first and settle the item deterministically.

4. Aggregate

Wins, ties and losses are combined per candidate and per workload category, checked against the Monitor's decision contract, and priced from dated catalog prices. The result is a recommendation — switch, route, or hold — with the evidence strength behind it. Savings are a 0.5×–2× range stamped with the date the prices were read, and are absent rather than zero when they cannot be resolved.

5. Verdict

Runs roll into the Monitor's living verdict. Each Run is a receipt; the verdict is what the Monitor currently believes. It goes out through Slack, a signed webhook, or back onto the original trace — see Where verdicts go.

Retries

Every phase is idempotent. A retried Run skips the responses and judgments it already has rather than generating them again, so a transient provider failure never doubles what a Run costs.