Tracing

tracing-crate span instrumentation for your workflows.

Behind the tracing feature gate (features = ["tracing"]).

Cano provides comprehensive observability through the optional tracing feature using the tracing library. All tracing instrumentation is behind conditional compilation, so it adds zero overhead when disabled.

For a callback-style API — get notified on workflow lifecycle and failure events without depending on the tracing ecosystem — see Observers. The tracing feature also ships a ready-made TracingObserver that bridges the two: attach it with .with_observer(Arc::new(TracingObserver::new())) to re-emit those observer events as tracing events under the cano::observer target.

#Setup

Enable the tracing feature flag in your Cargo.toml. You can also use features = ["all"] to enable everything (scheduler + tracing + recovery + metrics) at once.

[dependencies]
cano = { version = "0.13", features = ["tracing"] }

tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }

# Or enable everything (scheduler + tracing + recovery + metrics):
# cano = { version = "0.13", features = ["all"] }

#Basic Initialization

For quick setup during development, use the default formatter.

use tracing_subscriber;

// Simple setup for development
tracing_subscriber::fmt::init();

#Production Setup with Environment Filter

For production use, configure an environment filter so you can control log levels at runtime via the RUST_LOG environment variable.

use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};

// Production setup with env filter
tracing_subscriber::registry()
    .with(tracing_subscriber::EnvFilter::try_from_default_env()
        .unwrap_or_else(|_| "info".into()))
    .with(tracing_subscriber::fmt::layer())
    .init();

#What Gets Traced

#Error Tracing

When the tracing feature is enabled, Cano automatically instruments error paths at appropriate severity levels so you can diagnose failures without adding custom logging.

WARN task_attempt{attempt=2 max_attempts=4}: Task execution failed, will retry error="connection timeout"

ERROR task_attempt{attempt=4 max_attempts=4}: Task execution failed after all retry attempts error="connection timeout"

INFO workflow_orchestrate: Starting workflow execution initial_state=FetchData
ERROR workflow_orchestrate: Task failed in state FetchData after exhausting retries

#Filtering Trace Output

Use the RUST_LOG environment variable to control which modules emit trace output. This is especially useful in production to reduce noise.

# Show only Cano debug logs
RUST_LOG=cano=debug cargo run

# Show Cano info + your app's debug logs
RUST_LOG=cano=info,my_app=debug cargo run

# Show retry-related details only
RUST_LOG=cano::task=debug cargo run

# Silence everything except errors
RUST_LOG=error cargo run

#Scheduler Tracing

When both the scheduler and tracing features are enabled, the scheduler produces trace output for workflow lifecycle events. You can enable both with features = ["all"].

# Enable everything: scheduler + tracing + recovery + metrics
[dependencies]
cano = { version = "0.13", features = ["all"] }

#Custom Spans

Attach custom tracing spans to your workflows to include business-specific context. The span wraps all trace output generated during that workflow's execution.

use tracing::info_span;
use cano::prelude::*;

#[derive(Debug, Clone, PartialEq, Eq, Hash)]
enum MyState { Start, Complete }

#[derive(Clone)]
struct MyProcessor;

#[task(state = MyState)]
impl MyProcessor {
    async fn run_bare(&self) -> Result<TaskResult<MyState>, CanoError> {
        Ok(TaskResult::Single(MyState::Complete))
    }
}

fn build_workflow() -> Workflow<MyState> {
    // Create workflow with custom tracing span
    let workflow_span = info_span!(
        "user_data_processing",
        user_id = "12345",
        batch_id = "batch_001"
    );

    let store = MemoryStore::new();
    Workflow::new(Resources::new().insert("store", store))
        .with_tracing_span(workflow_span)
        .register(MyState::Start, MyProcessor)
        .add_exit_state(MyState::Complete)
}

#Custom Instrumentation

You can add your own tracing to Task implementations using the tracing crate's macros directly. This is useful for adding domain-specific context that Cano's built-in instrumentation does not cover.

#Instrumenting a Task

use cano::prelude::*;
use tracing::{info, warn};

#[derive(Debug, Clone, PartialEq, Eq, Hash)]
enum AppState { Process, Confirm }

async fn process_payment(_order_id: &str) -> Result<String, CanoError> {
    Ok("receipt-1234".to_string())
}

#[derive(Clone)]
struct PaymentTask;

#[task(state = AppState)]
impl PaymentTask {
    async fn run(&self, res: &Resources) -> Result<TaskResult<AppState>, CanoError> {
        let store = res.get::<MemoryStore, _>("store")?;
        let order_id: String = store.get("order_id")?;
        info!(order_id = %order_id, "Processing payment");

        match process_payment(&order_id).await {
            Ok(receipt) => {
                info!(order_id = %order_id, receipt = %receipt, "Payment succeeded");
                store.put("receipt", receipt)?;
                Ok(TaskResult::Single(AppState::Confirm))
            }
            Err(e) => {
                warn!(order_id = %order_id, error = %e, "Payment failed");
                Err(CanoError::task_execution(format!("payment failed: {e}")))
            }
        }
    }
}

#Correlating with Metrics

When the metrics feature is also enabled, tracing spans and metrics interoperate through the spans themselves. With the metrics-tracing-context crate wired in — a TracingContextLayer around your metrics recorder plus a MetricsLayer on your tracing-subscriber registry — every cano_* metric emitted while a span is entered inherits that span's fields as labels.

So a custom span you attach with with_tracing_span (or simply open around orchestrate()) flows its fields — a request_id, a tenant, … — onto the metrics recorded during that run. Cano's own default workflow_orchestrate and workflow_resume spans carry a workflow_id field whenever one is set via with_workflow_id, so it shows up both in the trace output and as a metric label.

#Example Output

Running with RUST_LOG=info produces structured logs:

INFO user_data_processing{user_id="12345"}: Starting workflow orchestration
  INFO user_data_processing{user_id="12345"}:task_execution: Starting task execution
    INFO user_data_processing{user_id="12345"}:task_attempt{attempt=1}: Task execution completed success=true
    INFO user_data_processing{user_id="12345"}:task_attempt{attempt=1}: processor_id=basic_processor input_records=3: Data processing completed
  INFO user_data_processing{user_id="12345"}: Workflow completed successfully

#Full Example

use cano::prelude::*;
use tracing::{info, info_span};
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};

#[derive(Debug, Clone, PartialEq, Eq, Hash)]
enum State {
    Start,
    Complete,
}

#[derive(Clone)]
struct ProcessOrder;

#[task(state = State)]
impl ProcessOrder {
    async fn run(&self, _res: &Resources) -> Result<TaskResult<State>, CanoError> {
        info!("Processing order...");
        Ok(TaskResult::Single(State::Complete))
    }
}

#[tokio::main]
async fn main() -> Result<(), CanoError> {
    // 1. Setup Subscriber with env filter
    tracing_subscriber::registry()
        .with(tracing_subscriber::EnvFilter::try_from_default_env()
            .unwrap_or_else(|_| "info".into()))
        .with(tracing_subscriber::fmt::layer())
        .init();

    let store = MemoryStore::new();

    // 2. Create Workflow with Custom Span
    // This span will wrap all logs generated by this workflow
    let workflow_span = info_span!(
        "order_processing",
        order_id = "ORD-2025-001",
        customer = "Acme Corp"
    );

    let workflow = Workflow::new(Resources::new().insert("store", store))
        .register(State::Start, ProcessOrder)
        .add_exit_state(State::Complete)
        .with_tracing_span(workflow_span); // Attach span

    // 3. Run
    info!("Submitting order...");
    workflow.orchestrate(State::Start).await?;

    Ok(())
}