Auto-Run (CI/CD for LLMs)
Automatically re-evaluate when models update or on a schedule.
Auto-Run is continuous evaluation — like CI/CD for your LLM stack. Configure triggers and PeerLM will automatically re-run your suite.
Triggers
- Model version update — when a provider releases a new version of a model in your suite, the old version is swapped for the new one and the suite runs automatically
- New model in tier — when a new model appears in a tier your suite uses, it's either auto-added or staged for your approval
- Schedule — run on a recurring cadence: Weekly (Mondays 6am UTC), Monthly (1st of month), or Quarterly
Staged Model Approval
When Auto-add new models is disabled, new models are staged for review. You'll see them on the suite detail page and can approve or dismiss each one. Approving triggers an auto-run.
Cost Controls
- Per-run credit cap — maximum credits an auto-run can consume. The run is skipped if it would exceed the cap.
- Monthly run limit — maximum auto-runs per month. Manual runs are not counted.
Notifications
Enable email notifications to be alerted when auto-runs complete. Add additional recipients (e.g., team leads) who should be notified.