Overview
Continuous comparison: ingest traffic, run comparisons, living switch/route verdict.
A Monitor connects PeerLM to a production use case and accumulates a series of comparison Runs against live traffic. Each Run samples prompts (± system prompts and control outputs), replays candidates on the same inputs, and judges quality retained vs your control alongside cost. The Monitor rolls those Runs into a living switch / route / hold verdict.
One-off authored evaluations stay under Suites and Runs in the sidebar—those remain a growth path for shareable head-to-head reports. Monitors are for continuous comparison from production traffic.
Core concepts
- Monitor — named production use case; owns control, candidates, sources, schedule, and the living verdict.
- Traffic — ingested records that become the Suite sampled at Run time.
- Run — one head-to-head batch (same primitive name as Suite Runs). Receipts live on the Run; the verdict lives on the Monitor.
- Recommendation — switch / route / hold, refreshed by each Run.
How it works
- Ingest — file, SDK, or connector ships traffic into the Monitor.
- Compare — a Run samples traffic, replays candidates, judges vs control.
- Verdict — the Monitor updates quality retained, savings, and routing from the growing pile of varied Runs.
What we optimize for
Volume and variety of head-to-heads—so you can decide whether a cheaper (or better) model holds up, and which task slices to route. We chart deltas vs control across Runs, not absolute quality scores, so shifting traffic mix does not fake quality movement. Drift watching is not the wedge; credible switch / route proof is.
Record contract
| Field | Required | Role |
|---|---|---|
prompt | Yes | User / task input to replay |
systemPrompt | Recommended | System context for faithful replay |
response | Recommended | Control output for parity judging |
model | Recommended | Labels the control; cost baseline |
tokensIn / tokensOut / latencyMs | Optional | Cost and latency baselines |
How to read the Monitor
- Living verdict — switch / route / hold as of the latest Run, with quality retained and projected savings.
- Trends — parity retained and savings deltas across Runs.
- Routing — per-task-slice recommendations from stratified comparisons.
- Run receipts — loss examples and methodology for one batch when you dig in.