Rate Limiting

Pace or shed calls to a rate-sensitive dependency.

This page covers the rate limiter in depth. See Resilience for how it composes with the other primitives, and Circuit Breakers for the related "dependency is down" case.

---

A circuit breaker stops calls to a dependency that's down. A RateLimiter paces calls to a dependency that's up but rate-sensitive — a third-party API with a per-second quota, a database, an LLM endpoint billed per request. It smooths bursty traffic into a steady rate the downstream can absorb.

It's a token bucket: a bucket holds fractional tokens; each acquisition spends one; tokens replenish at a fixed rate up to a capacity. Refill is lazy — there's no background task. Every acquire reads a single Instant, adds elapsed × refill_per_sec tokens (capped at capacity), then decides. A workflow that never builds a limiter pays nothing. The bucket starts full, so a burst of up to capacity calls is admitted instantly before sustained traffic settles to the refill rate.

Like a breaker, a limiter is cheap to clone (it's an Arc inside) — share one Arc<RateLimiter> across every task that draws on the same quota so the budget is enforced globally, including across tasks running in parallel inside a split/join state. Internally it's a synchronous parking_lot::Mutex with no awaits held across the critical section.

#RateLimiterPolicy

Build a policy with RateLimiterPolicy::per_second(n) (or ::new(tokens, period) for an arbitrary window) and tune it with the with_max_tokens / with_burst builders. Total bucket capacity is max_tokens + burst.

Field	Type	Meaning
max_tokens	u32	Steady-state bucket ceiling — and the size of the instantaneous burst a fresh limiter admits, since the bucket starts full. Defaults to tokens (one period's worth).
tokens_per_period	u32	Tokens added per refill_period.
refill_period	Duration	How long it takes to add tokens_per_period tokens. per_second(n) sets this to one second.
burst	u32	Extra capacity above max_tokens for short spikes. Defaults to 0.

RateLimiter::new panics on a misconfigured policy at construction: max_tokens == 0 (a zero-capacity bucket could never admit a call) or a zero refill rate (tokens_per_period == 0 or a zero refill_period — the bucket would never replenish). Both are programmer errors, caught before any task runs.

#Acquiring: try_acquire / acquire

• try_acquire() -> Option<Permit> — non-blocking. Some if a token was available (and consumes it), None if the bucket is empty. Use it to shed load.
• acquire().await -> Permit — if the bucket is empty it computes exactly how long until the next token refills, tokio::time::sleeps that long, and retries. Use it to pace work.

The returned Permit is a lightweight RAII marker for the call's scope. Unlike a circuit-breaker permit (which records a success/failure outcome) or a semaphore permit (which returns capacity on drop), a token-bucket permit's drop is a no-op — the token was already spent at acquisition and the bucket refills on the clock, not on release.

use cano::prelude::*;
use std::sync::Arc;

#[derive(Debug, Clone, PartialEq, Eq, Hash)]
enum Step { Call, Done }

#[derive(Clone)]
struct CallUpstream { limiter: Arc<RateLimiter> }

#[task(state = Step)]
impl CallUpstream {
    async fn run_bare(&self) -> Result<TaskResult<Step>, CanoError> {
        // Park until the shared budget admits this call, then proceed.
        let _permit = self.limiter.acquire().await;
        // ... call the rate-sensitive dependency ...
        Ok(TaskResult::Single(Step::Done))
    }
}

// 20 req/s, shared across every task that constructs from this Arc.
let limiter = Arc::new(RateLimiter::new(RateLimiterPolicy::per_second(20)));
let workflow = Workflow::bare()
    .register(Step::Call, CallUpstream { limiter: Arc::clone(&limiter) })
    .add_exit_state(Step::Done);

RateLimiter also implements Resource (no-op lifecycle), so instead of threading the Arc into each task you can register it once in Resources and look it up by key inside the task body — handy when several tasks share one quota.

#Token bucket vs fixed window

The token bucket is a faithful governor — it keeps you under a long-run rate and smooths bursts — but it is not a faithful model of a "resets-at-a-boundary" quota like a usage quota's "N per 5 hours, resets at 14:00." It drips capacity back continuously and has no reset instant to display. When you need that shape, use WindowedRateLimiter: a fixed-window counter that admits the full quota at once, resets as a step at the boundary, and exposes used() / remaining() / resets_at(). It resets lazily (no background task) and, like the bucket, is cheap to clone and implements Resource.

	RateLimiter (token bucket)	WindowedRateLimiter (fixed window)
Replenishment	continuous drip at the refill rate	step reset at the boundary
After exhaustion	one more unit every period/quota	zero until the reset, then the full quota
resets_at	none (boundary-less)	a displayable instant
Best for	pacing outbound load under a rate	mirroring a quota with a reset time

#Weighted cost

Both limiters meter weighted units: try_acquire_n(cost) / acquire_n(cost) consume cost units instead of one (the no-argument try_acquire / acquire are _n(1)). A request-count limit uses cost = 1; a usage/token budget uses the call's cost (e.g. 1500 tokens). tokens_available() / time_until(cost) expose the live state for observability and retry-after.

#Multi-level limiting (several tiers at once)

Real-world API limits often stack: a 5-hour cap and a weekly cap and a separate weekly cap for a single endpoint. MultiRateLimiter enforces them together — a request is admitted only if every applicable tier has room. Each tier is any Meter (a RateLimiter or a WindowedRateLimiter, mixed freely) with its own cost, so a request-count tier and a token-budget tier can share one gate.

The acquisition is atomic with no leak: it reserves each tier in turn, and if any tier rejects it drops the reservations gathered so far — refunding their units — so a partially-passing attempt never burns budget on the tiers that admitted it. (This is why a Reservation's drop refunds, unlike a committed Permit.) At most one tier's lock is held at a time, so there is no deadlock. On rejection it reports which tier blocked and the retry-after, as CanoError::RateLimited { tier, retry_after }.

use cano::prelude::*;
use std::sync::Arc;
use std::time::Duration;

let five_hour: Arc<dyn Meter> =
    Arc::new(WindowedRateLimiter::new(WindowPolicy::per_hours(500, 5)));
let weekly: Arc<dyn Meter> =
    Arc::new(WindowedRateLimiter::new(WindowPolicy::per_days(5_000, 7)));
let opus_weekly: Arc<dyn Meter> =
    Arc::new(WindowedRateLimiter::new(WindowPolicy::per_days(200, 7)));
// A usage/token budget metered in tokens, smoothed by a bucket.
let tokens: Arc<dyn Meter> = Arc::new(RateLimiter::new(
    RateLimiterPolicy::new(1_000_000, Duration::from_secs(60)).with_max_tokens(1_000_000),
));

let limiter = MultiRateLimiter::new()
    .with_tier("5h", five_hour, 1)
    .with_tier("weekly", weekly, 1)
    .with_tier("opus_weekly", opus_weekly, 1)
    .with_tier("tokens", tokens, 1500); // this call costs 1500 tokens

// Shed-load: which tier blocked, and for how long?
match limiter.try_acquire() {
    Ok(_permit) => { /* all tiers had room; proceed */ }
    Err(CanoError::RateLimited { tier, retry_after }) => {
        eprintln!("blocked by `{tier}`, retry after {retry_after:?}");
    }
    Err(_) => unreachable!(),
}

For a per-request subset — e.g. a non-Opus request that should skip the model-scoped tier — use try_acquire_for(&["5h", "weekly", "tokens"]) (or the async acquire_for). A tier with cost = 0 is inert (never blocks, never debited), another way to disable one conditionally.