Configuring Evaluation Criteria

Tell the judges what matters for this workload.

Criteria tell the judge panel what matters for this Monitor. Judges use them while making one pairwise choice between the control and candidate; they do not assign a separate 1–10 score to every criterion.

Default criteria

New Monitor Runs start with four comparison criteria:

Prompt adherence — follows the task and instructions
Absence of hallucination — avoids unsupported claims
Task completion — completes the requested work
Format adherence — respects the required output shape

Custom criteria

Pro, Team, and Enterprise can edit the labels, descriptions, and relative weights. A clear description matters because it is included in the judge prompt. The weight expresses importance to the pairwise decision; it does not create a weighted public score.

Write a testable instruction: "The answer cites only facts present in the supplied policy" is more useful than "The answer is good."

Objective validators

Use validators when the requirement can be checked without an LLM: normalized exact text, required RFC 6901 JSON paths, JSON Schema, or a safe regular expression. Validator failures are deterministic candidate failures. Validators never run arbitrary code, tools, or tests.

Mark a validator set complete only when passing every validator fully establishes parity for the item. Otherwise, passing validators leave the pair for the judge panel.