Resilience

Recover from transient faults — retries, timeouts, circuit breakers, rate limiting, bulkheads, panic safety, scheduler backoff.

"Resilient" in Cano's tagline isn't a vibe — it's a set of concrete, composable primitives that sit on the FSM dispatch path. Every one of them is opt-in: a workflow that wires none of them up pays nothing, and the hot path stays allocation-light. This is the guide: the circuit breaker and rate limiter get their full treatment here; retries, per-attempt timeouts, and bulkheads get an overview with a pointer to the API reference page that owns the builder methods.

The self-healing half of the tagline — checkpoint + resume, sagas, observers, health probes — lives in Recovery, Saga, and Observers.

---

#Retries

A task's config() returns a TaskConfig whose retry_mode drives the workflow dispatcher's retry loop. The default is exponential backoff with jitter (3 retries, 100ms base, 2.0× multiplier, 30s cap, 0.1 jitter); RetryMode also has None and Fixed(count, delay). On exhaustion the loop returns CanoError::RetryExhausted wrapping the last error.

use cano::prelude::*;
use std::time::Duration;

#[derive(Debug, Clone, PartialEq, Eq, Hash)]
enum Step { Fetch, Done }

#[derive(Clone)]
struct FetchTask;

#[task(state = Step)]
impl FetchTask {
    fn config(&self) -> TaskConfig {
        // 5 attempts, 200ms apart; or `.with_exponential_retry(n)` / `TaskConfig::minimal()` for none.
        TaskConfig::new().with_fixed_retry(4, Duration::from_millis(200))
    }
    async fn run_bare(&self) -> Result<TaskResult<Step>, CanoError> {
        // ... call the flaky thing ...
        Ok(TaskResult::Single(Step::Done))
    }
}

See Tasks → Configuration & Retries for the full RetryMode / TaskConfig reference.

---

#Per-Attempt Timeouts

TaskConfig::with_attempt_timeout(Duration) wraps each retry attempt in tokio::time::timeout. A blown deadline produces CanoError::Timeout, which the retry loop treats as a recoverable failure — the attempt is retried like any other, and only if every attempt times out does the final timeout get wrapped in CanoError::RetryExhausted. Combine it with retries to bound total wall-clock per state: 3 attempts × a 2s attempt timeout ≈ 6s worst case (plus backoff).

#[task(state = Step)]
impl CallTask {
    fn config(&self) -> TaskConfig {
        TaskConfig::new()
            .with_fixed_retry(2, Duration::from_millis(100))
            .with_attempt_timeout(Duration::from_secs(2)) // each attempt gets 2s
    }
    async fn run_bare(&self) -> Result<TaskResult<Step>, CanoError> {
        Ok(TaskResult::Single(Step::Done))
    }
}

Distinct from Workflow::with_total_timeout (the wall-clock budget for the entire orchestration — see below) and from the legacy Workflow::with_timeout (a blunt outer tokio::time::timeout with no graceful compensation). The full TaskConfig / RetryMode API — including how attempt timeouts compose with each retry mode — lives in Tasks → Configuration & Retries.

---

#Workflow Total Timeout

Workflow::with_total_timeout(Duration) sets a wall-clock budget for the entire orchestrate (or resume_from) call. When the budget elapses, the in-flight task is aborted at its next await point, the saga compensation stack drains against its own bounded budget, and the call returns CanoError::WorkflowTimeout { elapsed, limit } wrapped in CanoError::WithStateContext (or CanoError::CompensationFailed if any compensate also fails).

let workflow = Workflow::bare()
    .with_total_timeout(Duration::from_millis(200))
    // Optional override; defaults to min(remaining_budget / 2, 30s).
    .with_compensation_timeout(Duration::from_millis(50))
    .register_with_compensation(Step::Reserve, Reserve)
    .register_with_compensation(Step::Charge, Charge)
    .register(Step::Ship, Ship)
    .add_exit_state(Step::Done);

The budget compounds with every other resilience primitive — retries, per-attempt timeouts, circuit breakers — because they all run inside the per-state dispatch the engine wraps in tokio::time::timeout_at. A task that retries forever doesn't outlive the total budget.

#Compensation drain budget

When a total-timeout trip drains the compensation stack, each compensate call is wrapped in tokio::time::timeout_at against a derived deadline. The default is min(remaining_budget / 2, 30s) — half of whatever budget is left when the timeout fires, capped at 30 seconds (and 30s as the floor when nothing is left). Set Workflow::with_compensation_timeout(Duration) to override this explicitly. A compensate that exceeds the deadline is recorded as a timeout error and the drain continues — remaining entries error fast under the now-elapsed deadline rather than extending the rollback indefinitely.

#Observer hook

A WorkflowObserver attached via with_observer receives one on_workflow_timeout(elapsed, limit) call when the budget fires, before the compensation drain runs. With the tracing feature, TracingObserver re-emits it as a WARN-level "workflow total timeout exceeded" event with elapsed_ms / limit_ms fields under the cano::observer target. See Observers → Lifecycle Events for the full hook reference.

#The three timeout knobs

API	Scope	On expiry	Compensation drain
TaskConfig::with_attempt_timeout	One attempt of one task	CanoError::Timeout — retried like any other failure; final timeout becomes RetryExhausted	Triggered like any other terminal task error (unbounded)
Workflow::with_total_timeout	The entire orchestrate / resume_from call	In-flight task aborted; CanoError::WorkflowTimeout (wrapped in WithStateContext)	Bounded by with_compensation_timeout or the default min(remaining/2, 30s)
Workflow::with_timeout (legacy)	The whole orchestration future	CanoError::Workflow("Workflow timeout exceeded") — no graceful abort	None — the future is dropped abruptly

Pick with_total_timeout for any new code that needs a workflow-wide budget. The legacy with_timeout remains for backward compatibility and composes naturally — if both are set, whichever fires first wins.

---

#Circuit Breakers & Rate Limiting

Two of the most-used resilience primitives have their own dedicated pages:

• Circuit Breakers — stop calling a dependency that is down, before the retry loop even runs.
• Rate Limiting — pace or shed calls to a rate-sensitive dependency, including multi-tier limits.

---

#Bulkheads (split concurrency)

A split/join state runs its tasks in parallel — but "parallel" can mean "500 connections to a database that handles 50". JoinConfig::with_bulkhead(n) gates the task bodies on a shared Semaphore so at most n run concurrently; the rest queue. Tasks still all get spawned and the join strategy is unchanged — only their execution is rate-limited. Full details on the Split & Join page.

let join = JoinConfig::new(JoinStrategy::All, S::Join).with_bulkhead(8); // ≤ 8 at a time
let workflow = Workflow::bare()
    .register_split(S::Fan, (0..200).map(|_| W).collect(), join)
    .add_exit_state(S::Done);

---

#Panic Safety

A panic! inside a task body — or a `.unwrap()` that fired — does not unwind through the workflow engine and abort the runtime worker. The dispatcher wraps each task in catch_unwind; a panic becomes CanoError::TaskExecution("panic: …"), which then flows through the normal retry / error / (with a saga) compensation path. Split tasks do the same per spawned task, so a panic preserves its task index. There's nothing to configure — it's always on.

---

#Scheduler Backoff & Trip

Behind the scheduler feature gate.

The primitives above act per dispatch. When a workflow runs on a timer there's a second, flow-level layer: a per-flow BackoffPolicy stretches the gap between failed runs and can trip a flow that keeps failing so it stops firing until RunningScheduler::reset_flow(id) clears it — surfaced as Status::Backoff { … } / Status::Tripped { … }. That's documented on its own page: Scheduler → Backoff & Trip State (runnable demo: cargo run --example scheduler_backoff --features scheduler).

---

#Composing It All

These stack cleanly. A typical "talk to a flaky external service" state:

• Circuit breaker (shared across every task hitting that service) — bail fast when it's down.
• Rate limiter (also shared) — stay under the service's quota when it's up.
• Per-attempt timeout — don't hang on a slow call.
• Retries with backoff — ride out transients.
• If the call mutated something, make it a compensatable task so a downstream failure can undo it.
• Attach a checkpoint store so a crash mid-workflow doesn't lose progress.
• Attach a WorkflowObserver (or just TracingObserver) to see retries, circuit-opens, and checkpoints as they happen.

The circuit_breaker, workflow_recovery, saga_payment, and workflow_observer / observer_metrics examples shipped with the crate each exercise one of these in isolation.