Scheduler Backoff & Trip State

What happens when a scheduled flow keeps failing — and how to recover it.

See Scheduler for scheduling strategies and lifecycle. This page covers what the scheduler does when a flow fails repeatedly.

A flow that fails repeatedly shouldn't keep re-firing on its base schedule, so every flow has a BackoffPolicy. After a failure the scheduler parks the flow in Status::Backoff for a growing delay; with a streak_limit set it eventually Trippeds and stops dispatching until you intervene. Each flow starts with BackoffPolicy::default() — call set_backoff to use a different one.

#Overriding the Default Policy

Register the workflow normally, then call set_backoff before start(). The policy controls the initial delay after the first failure, the multiplier applied per additional consecutive failure, a hard cap on the computed delay, jitter, and an optional streak limit.

use cano::prelude::*;
use std::time::Duration;

#[derive(Debug, Clone, PartialEq, Eq, Hash)]
enum FlowState { Start, Done }

#[derive(Clone)]
struct NoopTask;

#[task(state = FlowState)]
impl NoopTask {
    async fn run_bare(&self) -> Result<TaskResult<FlowState>, CanoError> {
        Ok(TaskResult::Single(FlowState::Done))
    }
}

#[tokio::main]
async fn main() -> Result<(), CanoError> {
    let mut scheduler: Scheduler<FlowState> = Scheduler::new();

    let workflow = Workflow::new(Resources::new())
        .register(FlowState::Start, NoopTask)
        .add_exit_state(FlowState::Done);

    scheduler.every(
        "flaky",
        workflow,
        FlowState::Start,
        Duration::from_millis(200),
    )?;

    scheduler.set_backoff(
        "flaky",
        BackoffPolicy {
            initial: Duration::from_millis(300),
            multiplier: 2.0,
            max_delay: Duration::from_secs(2),
            jitter: 0.1,
            streak_limit: Some(3),
        },
    )?;

    let running = scheduler.start().await?;
    running.wait().await?;
    Ok(())
}

Computed delay is initial * multiplier^(streak-1), capped at max_delay, then multiplied by a random factor in 1 ± jitter. The Every loop's sleep extends to max(interval, next_eligible - now), and the Cron loop suppresses ticks inside the backoff window. BackoffPolicy::default() gives 1s initial, 2.0× multiplier, 5min cap, 0.1 jitter, and no trip limit. Use BackoffPolicy::with_trip(n) to ask for a trip after n consecutive failures.

#Status Variants

Status is #[non_exhaustive] — external match arms must include a wildcard. The variants are:

• Idle — registered, never run or finished cleanly.
• Running — currently executing.
• Completed — last run reached an exit state.
• Backoff { until, streak, last_error } — last run errored; the flow is waiting until until before its next dispatch, per its BackoffPolicy.
• Tripped { streak, last_error } — streak reached streak_limit; the scheduler will not dispatch this flow again until reset_flow is called.

Outcome writes are atomic: a single write decides this run's terminal status (Completed on success, otherwise Backoff or Tripped), so observers never see a transient intermediate state. FlowInfo exposes failure_streak and next_eligible for observability.

#Recovery via reset_flow

A Tripped flow stays parked until you clear it. RunningScheduler::reset_flow(id) clears the failure streak and next_eligible, and (when the flow is not currently running) sets the status back to Idle. Manual trigger() is rejected on a tripped flow — call reset_flow first.

let snap = running.status("flaky").await.expect("flow exists");
if matches!(snap.status, Status::Tripped { .. }) {
    running.reset_flow("flaky").await?;
}

See the scheduler_backoff example (cargo run --example scheduler_backoff --features scheduler) for an end-to-end walk-through that exercises the trip and recovery path.