Promotion & drift

How a step earns autonomy, and how the platform keeps it honest.

An L2 step (automation + approval) can be promoted to L1 (autonomous), but only with evidence, and only while the platform keeps watching it.

Promotion is not automatic today. A step is promoted by setting its promotion evidence in the workflow definition (a configuration change), and no step ships pre-promoted. The drift machinery below is implemented but only applies once a step has been promoted.

Earning promotion

Promotion requires evidence that the step is reliable:

  • A scenario eval batch passing at n ≥ 100.
  • A corpus replay of at least 30 previously approved cases.
  • Both below a maximum failure-rate threshold.

The promotion record captures the prompt, model, and served model at the time, so "promoted on model X" is auditable.

Spot-checks

A promoted step isn't left unwatched. A fraction of its L1 outputs (its spot-check rate) are routed back for after-the-fact human review. Spot-check acceptance is tracked and feeds back into the step's health.

The spot-check rate is configured separately from promotion. It is the spot_check_rate field on the step's JobTemplate, and it defaults to 0.0 — promotion evidence alone does not turn sampling on. A promoted step must set spot_check_rate > 0 (enforced at definition time), otherwise no spot checks are created and the queue's Spot checks tab stays empty.
In the shipped workflows today, no step is promoted and no template sets spot_check_rate, so live runs generate no spot checks at all. The only exception is seeded demo data: the demo seeder fabricates spot checks at its own 20% rate (seed_demo.py), independent of the workflow definitions, so a seeded database can populate the tab.

Drift detection

The platform re-checks whether a promotion still holds. A step is flagged as drifted if:

  • Its prompt changed since promotion.
  • Its model changed since promotion.
  • Its spot-check acceptance fell below the minimum.

A drifted step falls back to L2: it keeps running, but a human must approve it again until the drift is resolved.

This is the safety net: autonomy is earned, auditable, and reversible.