Cano v0.15.2

Resilience

Recover from transient faults — retries, timeouts, circuit breakers, rate limiting, bulkheads, panic safety, scheduler backoff.

"Resilient" in Cano's tagline isn't a vibe — it's a set of concrete, composable primitives that sit on the FSM dispatch path. Every one of them is opt-in: a workflow that wires none of them up pays nothing, and the hot path stays allocation-light. This is the guide: the circuit breaker and rate limiter get their full treatment here; retries, per-attempt timeouts, and bulkheads get an overview with a pointer to the API reference page that owns the builder methods.

The self-healing half of the tagline — checkpoint + resume, sagas, observers, health probes — lives in Recovery, Saga, and Observers.

Retries

A task's config() returns a TaskConfig whose retry_mode drives the workflow dispatcher's retry loop. The default is exponential backoff with jitter (3 retries, 100ms base, 2.0× multiplier, 30s cap, 0.1 jitter); RetryMode also has None and Fixed(count, delay). On exhaustion the loop returns CanoError::RetryExhausted wrapping the last error. 

use cano::prelude::*;
use std::time::Duration;

#[derive(Debug, Clone, PartialEq, Eq, Hash)]
enum Step { Fetch, Done }

#[derive(Clone)]
struct FetchTask;

#[task(state = Step)]
impl FetchTask {
    fn config(&self) -> TaskConfig {
        // 5 attempts, 200ms apart; or `.with_exponential_retry(n)` / `TaskConfig::minimal()` for none.
        TaskConfig::new().with_fixed_retry(4, Duration::from_millis(200))
    }
    async fn run_bare(&self) -> Result<TaskResult<Step>, CanoError> {
        // ... call the flaky thing ...
        Ok(TaskResult::Single(Step::Done))
    }
}

See Tasks → Configuration & Retries for the full RetryMode / TaskConfig reference.

Per-Attempt Timeouts

TaskConfig::with_attempt_timeout(Duration) wraps each retry attempt in tokio::time::timeout. A blown deadline produces CanoError::Timeout, which the retry loop treats as a recoverable failure — the attempt is retried like any other, and only if every attempt times out does the final timeout get wrapped in CanoError::RetryExhausted. Combine it with retries to bound total wall-clock per state: 3 attempts × a 2s attempt timeout ≈ 6s worst case (plus backoff). 

#[task(state = Step)]
impl CallTask {
    fn config(&self) -> TaskConfig {
        TaskConfig::new()
            .with_fixed_retry(2, Duration::from_millis(100))
            .with_attempt_timeout(Duration::from_secs(2)) // each attempt gets 2s
    }
    async fn run_bare(&self) -> Result<TaskResult<Step>, CanoError> {
        Ok(TaskResult::Single(Step::Done))
    }
}

Distinct from Workflow::with_total_timeout (the wall-clock budget for the entire orchestration — see below). The full TaskConfig / RetryMode API — including how attempt timeouts compose with each retry mode — lives in Tasks → Configuration & Retries.

Workflow Total Timeout

Workflow::with_total_timeout(Duration) sets a wall-clock budget for the entire orchestrate (or resume_from) call. When the budget elapses, the in-flight task is aborted at its next await point, the saga compensation stack drains against its own bounded budget, and the call returns CanoError::WorkflowTimeout { elapsed, limit } wrapped in CanoError::WithStateContext (or CanoError::CompensationFailed if any compensate also fails). 

let workflow = Workflow::bare()
    .with_total_timeout(Duration::from_millis(200))
    // Optional override; defaults to min(remaining_budget / 2, 30s).
    .with_compensation_timeout(Duration::from_millis(50))
    .register_with_compensation(Step::Reserve, Reserve)
    .register_with_compensation(Step::Charge, Charge)
    .register(Step::Ship, Ship)
    .add_exit_state(Step::Done);

The budget compounds with every other resilience primitive — retries, per-attempt timeouts, circuit breakers — because they all run inside the per-state dispatch the engine wraps in tokio::time::timeout_at. A task that retries forever doesn't outlive the total budget.

Compensation drain budget

When a total-timeout trip drains the compensation stack, each compensate call is wrapped in tokio::time::timeout_at against a derived deadline. The default is min(remaining_budget / 2, 30s) — half of whatever budget is left when the timeout fires, capped at 30 seconds (and 30s as the floor when nothing is left). Set Workflow::with_compensation_timeout(Duration) to override this explicitly. A compensate that exceeds the deadline is recorded as a timeout error and the drain continues — remaining entries error fast under the now-elapsed deadline rather than extending the rollback indefinitely.

Observer hook

A WorkflowObserver attached via with_observer receives one on_workflow_timeout(elapsed, limit) call when the budget fires, before the compensation drain runs. With the tracing feature, TracingObserver re-emits it as a WARN-level "workflow total timeout exceeded" event with elapsed_ms / limit_ms fields under the cano::observer target. See Observers → Lifecycle Events for the full hook reference.

The two timeout knobs

Two knobs, two scopes — the total budget wins

A workflow with a 7 second total budget: State A finishes in 1s, then State B retries three times under a 2 second per-attempt timeout — attempts 1 and 2 each blow their own deadline and are retried, and the total budget fires at 7s, aborting attempt 3 mid-flight before its own 8s deadline could. WorkflowTimeout Total budget with_total_timeout(7s) Workflow Timeout Timeout attempt 3 cut short State A attempt 1 attempt 2 0s 1s 2s 3s 4s 5s 6s 7s 8s attempt_timeout(2s) fires twice and is retried; the 7s budget then aborts attempt 3 mid-flight the total budget wins — no retry policy can outlive it
APIScopeOn expiryCompensation drain
TaskConfig::with_attempt_timeoutOne attempt of one taskCanoError::Timeout — retried like any other failure; final timeout becomes RetryExhaustedTriggered like any other terminal task error (unbounded)
Workflow::with_total_timeoutThe entire orchestrate / resume_from callIn-flight task aborted; CanoError::WorkflowTimeout (wrapped in WithStateContext)Bounded by with_compensation_timeout or the default min(remaining/2, 30s)

Reach for with_attempt_timeout to bound a single call and with_total_timeout for a workflow-wide budget; they compose. To stop a run on an external signal rather than a deadline, use cooperative cancellation.

Runnable example: cargo run --example workflow_total_timeout — a 3-step saga where the shipping step overruns the budget; the timeout fires, the prior steps' compensations run in reverse, and the final error is the wrapped WorkflowTimeout.

Cooperative Cancellation

Where a total timeout aborts a run on a deadline, cancellation aborts it on a signal you control — a shutdown handler, a user "stop" button, a parent task giving up. Workflow::orchestrate(start, token) (and resume_from(id, token)) always take a CancellationToken; firing the paired CancellationHandle aborts the in-flight task at its next await point, drains the saga compensation stack, and returns CanoError::Cancelled wrapped in CanoError::WithStateContext (or CanoError::CompensationFailed if a compensate also fails). 

use cano::CancellationToken;

let (handle, token) = CancellationToken::new();

// Cancel from anywhere — a signal handler, a sibling task, a timer:
let canceller = tokio::spawn(async move {
    shutdown_signal().await;
    handle.cancel(); // idempotent; the handle is Clone, so many owners can trigger it
});

let result = workflow.orchestrate(Step::Reserve, token).await;
assert!(matches!(result, Err(e) if e.category() == "cancelled"));

To opt a run out of cancellation, pass CancellationToken::disabled() instead of a live token: workflow.orchestrate(start, CancellationToken::disabled()). A disabled token never fires and is zero-cost — the cancellation select! is skipped entirely.

Cancellation is cooperative. The engine drops the running task's future at its next .await. A task spinning in a tight synchronous/CPU loop with no .await is not interrupted until it next yields — design long-running task bodies to .await periodically if they must be cancellable.

Saga safety

A compensatable task is never interrupted mid-run. Aborting it after an in-task side effect committed but before its Output reached the compensation stack would orphan that side effect with nothing to roll back. So a CompensatableTask always runs to completion (recording its rollback entry); the cancellation is honoured at the next state boundary, which then drains the now-complete stack. The compensation drain itself is uncancellable — a cancel that lands during rollback does not abort the remaining compensators.

Observer hook & precedence

A WorkflowObserver receives one on_cancelled(state) call when the cancel is observed, before the drain runs; TracingObserver re-emits it as a WARN event and MetricsObserver increments cano_observed_cancellations_total. Against with_total_timeout, cancellation wins: it is checked deterministically at each state boundary and biased ahead of the per-state budget mid-task.

The scheduler builds on this: RunningScheduler::cancel_flow(id) cancels an in-flight scheduled run, and graceful stop() cancels every in-flight flow (rolling back their sagas) instead of waiting for them to finish.

Runnable example: cargo run --example workflow_cancellation — a 3-step saga where a sibling task cancels the shipping step mid-flight; the prior steps' compensations run in reverse and the final error is the wrapped Cancelled.

Circuit Breakers & Rate Limiting

Two of the most-used resilience primitives have their own dedicated pages:

  • Circuit Breakers — stop calling a dependency that is down, before the retry loop even runs.
  • Rate Limiting — pace or shed calls to a rate-sensitive dependency, including multi-tier limits.

Bulkheads (split concurrency)

A split/join state runs its tasks in parallel — but "parallel" can mean "500 connections to a database that handles 50". JoinConfig::with_bulkhead(n) gates the task bodies on a shared Semaphore so at most n run concurrently; the rest queue. Tasks still all get spawned and the join strategy is unchanged — only their execution is rate-limited. Full details on the Split & Join page. 

let join = JoinConfig::new(JoinStrategy::All, S::Join).with_bulkhead(8); // ≤ 8 at a time
let workflow = Workflow::bare()
    .register_split(S::Fan, (0..200).map(|_| W).collect(), join)
    .add_exit_state(S::Done);

Runnable example: cargo run --example split_bulkhead — 8 split tasks behind a with_bulkhead(2); the start/end timestamps make the rate-limiting visible.

Panic Safety

A panic! inside a task body — or a `.unwrap()` that fired — does not unwind through the workflow engine and abort the runtime worker. The dispatcher wraps each task in catch_unwind; a panic becomes CanoError::TaskExecution("panic: …"), which then flows through the normal retry / error / (with a saga) compensation path. Split tasks do the same per spawned task, so a panic preserves its task index. There's nothing to configure — it's always on.

Runnable example: cargo run --example panic_safety — a task that panic!s mid-workflow; the engine returns CanoError::TaskExecution("panic: …") and the preceding compensatable step's rollback still runs.

Scheduler Backoff & Trip

Behind the scheduler feature gate.

The primitives above act per dispatch. When a workflow runs on a timer there's a second, flow-level layer: a per-flow BackoffPolicy stretches the gap between failed runs and can trip a flow that keeps failing so it stops firing until RunningScheduler::reset_flow(id) clears it — surfaced as Status::Backoff { … } / Status::Tripped { … }. That's documented on its own page: Scheduler → Backoff & Trip State (runnable demo: cargo run --example scheduler_backoff --features scheduler).

Composing It All

These stack cleanly. A typical "talk to a flaky external service" state:

One dispatch, four nested layers — and the CanoError each one surfaces

One task dispatch passes through four nested layers: the workflow total timeout wraps the retry loop, the retry loop checks the circuit breaker before each attempt, and the per-attempt timeout wraps the attempt — your task body, where the rate limiter permit is taken before the upstream call. An open breaker leaves immediately with CircuitOpen; a Timeout or RateLimited is retried; exhausting the attempts returns RetryExhausted. Workflow total timeout timeout_at(start + limit) → WorkflowTimeout aborts mid-flight retry loop run_with_retries — attempts 1..max_attempts tokio::time::timeout(attempt_timeout) your task body Timeout · RateLimited · any task Err retried per RetryMode, then a backoff sleep success circuit breaker try_acquire() rate limiter acquire() upstream call the flaky thing CircuitOpen no attempt, no retry burned RetryExhausted after max_attempts Ok(next state) workflow advances
  • Circuit breaker (shared across every task hitting that service) — bail fast when it's down.
  • Rate limiter (also shared) — stay under the service's quota when it's up.
  • Per-attempt timeout — don't hang on a slow call.
  • Retries with backoff — ride out transients.
  • If the call mutated something, make it a compensatable task so a downstream failure can undo it.
  • Attach a checkpoint store so a crash mid-workflow doesn't lose progress.
  • Attach a WorkflowObserver (or just TracingObserver) to see retries, circuit-opens, and checkpoints as they happen.

The circuit_breaker, workflow_recovery, saga_payment, and workflow_observer / observer_metrics examples shipped with the crate each exercise one of these in isolation.

↑↓ to navigate ↵ to open esc to close