Evaluation Errors
Error categories, retryable vs non-retryable, and how to retry.
Candidate generation, deterministic validation, and judging are separate parts of a Monitor Run. The report identifies which part failed; missing telemetry is never counted as a model loss.
| Category | Meaning | Likely action |
|---|---|---|
| Provider or rate-limit error | Temporary upstream failure | Wait for the worker retry or start again after the provider recovers |
| Timeout | A call exceeded its execution limit | Check provider health and prompt size |
| Model unavailable | The catalog model cannot currently execute | Choose another candidate or wait for catalog sync |
| Context too long | The rendered request exceeds model capability | Shorten the input or choose a larger-context model |
| Content filter or refusal | The provider did not return a usable answer | Inspect the item before changing the prompt or candidate |
| Judge parse failure | A judge did not return the pairwise contract | PeerLM records the failed judgment; the panel may make the item indeterminate |
| Validator failure | A declarative requirement failed | Inspect the exact text, JSON path, schema, or regex result |
Judges return a winner, failure tag, and one sentence. There is no per-evaluator 1–10 score to retry or recompute.