Circuit Breakers

Stop calling a dependency that is down — before the retry loop even runs.

This page covers the circuit breaker in depth. See Resilience for how it composes with retries, timeouts, and the other primitives.

Retries help with transient faults; a CircuitBreaker helps when a dependency is down — it stops hammering it. Once it has seen failure_threshold consecutive failures the breaker opens: every subsequent call short-circuits with CanoError::CircuitOpen without invoking the task body and without consuming a retry attempt, so the unhealthy dependency gets a break to recover.

A breaker is cheap to clone (it's an Arc inside) — share one Arc<CircuitBreaker> across every task that hits the same dependency so a trip from any caller protects every caller, including tasks running in parallel inside a split/join state. Internally it's a synchronous std::sync::Mutex with no awaits held across the critical section, so concurrent acquires from split tasks are safe.

#The state machine

The state is Closed → Open { until } → HalfOpen → Closed:

• Closed — calls flow through; consecutive failures count toward failure_threshold. A success resets the counter.
• Open { until } — every try_acquire returns CanoError::CircuitOpen until the clock passes until (= when_it_opened + reset_timeout).
• HalfOpen — entered lazily: there is no background task; the first try_acquire after until performs the Open → HalfOpen transition. It admits up to half_open_max_calls trial calls; that many consecutive successes close the breaker, and any failure re-opens it with a fresh cool-down.

#CircuitPolicy

A breaker is constructed from a CircuitPolicy { failure_threshold, reset_timeout, half_open_max_calls }:

CircuitBreaker::new panics on a misconfigured policy at construction: half_open_max_calls == 0 would deadlock the breaker permanently in HalfOpen (no probe could ever be admitted), and values approaching u32::MAX would either saturate the success counter before reaching the threshold or take effectively forever to close. Both are programmer errors, caught before any task runs.

#Permits and the RAII API

The breaker's primitive operations are:

• try_acquire() -> Result<Permit, CanoError> — Err(CanoError::CircuitOpen) when the breaker is Open (or when HalfOpen has already handed out its half_open_max_calls trial permits).
• record_success(permit) — the call succeeded.
• record_failure(permit) — the call failed.

A Permit that is dropped without being consumed counts as a failure — so an early return, a ? bail-out, or a panic doesn't leave the breaker believing the call succeeded. That's the panic-safety guarantee for the manual path.

Permits also carry an epoch tag — a counter the breaker bumps on every state transition (Closed → Open, Open → HalfOpen, HalfOpen → Open, HalfOpen → Closed). When a permit is consumed (success, failure, or RAII drop) the breaker compares the permit's epoch against the current one and silently discards stale outcomes. This means a slow caller whose call straddles a state-machine session — e.g. a request that started in Closed and only returned after the breaker tripped, cooled down, and entered a fresh HalfOpen probe — cannot accidentally close the breaker on the strength of a result that was never meant to count as a probe.

#Wiring it in: TaskConfig::with_circuit_breaker

The common path — attach the breaker to a task's config and the workflow's retry loop does the rest. It consults the breaker before each attempt, records the outcome after the task body, and an open breaker short-circuits the whole retry loop: the CircuitOpen error is returned raw, never wrapped in RetryExhausted. A per-attempt timeout firing is recorded as a circuit failure too, so the breaker also guards against slow upstreams.

use cano::prelude::*;
use std::sync::Arc;
use std::time::Duration;

#[derive(Debug, Clone, PartialEq, Eq, Hash)]
enum Step { Call, Done }

#[derive(Clone)]
struct CallUpstream { breaker: Arc<CircuitBreaker> }

#[task(state = Step)]
impl CallUpstream {
    fn config(&self) -> TaskConfig {
        TaskConfig::new()
            .with_fixed_retry(2, Duration::from_millis(50))
            .with_circuit_breaker(Arc::clone(&self.breaker))
    }
    async fn run_bare(&self) -> Result<TaskResult<Step>, CanoError> {
        // ... call the dependency; an Err here counts against the breaker ...
        Ok(TaskResult::Single(Step::Done))
    }
}

let breaker = Arc::new(CircuitBreaker::new(CircuitPolicy {
    failure_threshold: 3,
    reset_timeout: Duration::from_secs(5),
    half_open_max_calls: 1,
}));
let workflow = Workflow::bare()
    .register(Step::Call, CallUpstream { breaker: Arc::clone(&breaker) })
    .add_exit_state(Step::Done);

#Driving it by hand: try_acquire / record_*

When wiring it through TaskConfig is awkward — e.g. you guard several distinct calls in one task body, or the breaker is shared with non-task code — register the breaker as a resource (it implements Resource with no-op lifecycle), look it up by key inside the task, and drive the RAII API directly:

let permit = breaker.try_acquire()?;        // Err(CanoError::CircuitOpen) when open
match do_the_call().await {
    Ok(v)  => { breaker.record_success(permit); Ok(v) }
    Err(e) => { breaker.record_failure(permit); Err(e) }
}

The trade-off: this path bypasses both the retry-loop short-circuit (you decide whether CircuitOpen aborts or retries) and the before-call / after-call ordering guarantee the with_circuit_breaker integration gives you for free. So prefer TaskConfig::with_circuit_breaker whenever a task maps one-to-one to one guarded call; reach for the manual path only when it genuinely doesn't.

Runnable demos: cargo run --example circuit_breaker (the TaskConfig::with_circuit_breaker integration path) and cargo run --example circuit_breaker_manual (the manual try_acquire / record_* path shown above — breaker opens after a failure streak, rejects calls while open, then closes via HalfOpen).