Local (in-process)

In one line

Runs a GGUF language model of a supported architecture (see Local models and formats) on the CPU or a GPU inside your process, with no server and no HTTP endpoint.

You need

The provider-local feature for the CPU, or provider-local-vulkan, provider-local-metal, provider-local-cuda or provider-local-rocm for a GPU. A GGUF file, or disk for the one-time download (0.29 GB to 8.03 GB across the presets, 2.84 GB for the default). Memory of about the file size plus the K/V cache.

Read this if

You are routing requests through ProviderEndpoint::Local, or you want a model that needs no external service.

This is the only provider without an HTTP endpoint. LocalModel::load resolves the GGUF file, loads the weights, the tokenizer and the chat template, and starts one engine thread. CucaClientBuilder::with_local_model attaches the model to a client built for ProviderEndpoint::Local. The engine code is pure Rust: no llama.cpp and no FFI written by cuca.

Endpoint

Each GPU feature enables provider-local and provider-local-gpu, the shared GPU engine, which selects no backend by itself. With the feature off, dispatching ProviderEndpoint::Local returns CucaError::ProviderNotEnabled("provider-local").

Authentication

There is no API key and no auth header. The one credential is the optional Hugging Face token (hf_token, else HF_TOKEN), sent as a bearer token on the model download. The pinned preset files download anonymously.

Entry points

• LocalModel::load(LocalModelConfig) is async. It resolves or downloads the file, loads it and starts the thread named cuca-local-engine. The download, the hashing and the weight parsing run on that thread, so the call never blocks the async runtime.
• CucaClientBuilder::with_local_model(LocalModel) attaches the model. CucaClient::local_model() returns Option<&LocalModel>.
• LocalModel is cheap to clone. Every clone and every client built from a clone shares the one engine thread, which ends when the last clone is dropped.
• LocalModel::kv_cache_bytes() returns the K/V cache size in bytes. LocalModel::capabilities() returns LocalCapabilities { reasoning, can_disable_thinking, tools }. LocalModel::draft_stats() returns the drafter counters.
• ModelSource::PRESETS, ModelSource::preset(name), ModelSource::default_model() and ModelSource::hugging_face(repo, revision, file) describe model sources. ModelSource and LocalDevice also implement FromStr.

Example

The smallest stream: the default model, the Local variant and every other builder default.

cargo add cuca --path /path/to/cuca --features provider-local
cargo add tokio --features rt,macros
cargo add tokio-stream

use std::io::{Write, stdout};

use cuca::types::{MessageContentBlock, ProviderEndpoint};
use cuca::{CucaClient, LocalModel, LocalModelConfig, UnifiedRequest};
use tokio_stream::StreamExt;

#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Downloads the pinned default model (Gemma 4 E2B Q4_0) on first use, then
    // loads it.
    let model = LocalModel::load(LocalModelConfig::default()).await?;

    let client = CucaClient::builder()
        .with_provider(ProviderEndpoint::Local)
        .with_local_model(model)
        .build()?;

    // The request's model name is not used for routing: one client holds one model.
    let request = UnifiedRequest::new("local")
        .add_system_message("You are concise.")
        .add_user_message("Explain CUCA in one sentence.")
        .set_max_tokens(256);
    let mut stream = client.generate_stream(request).await?;
    while let Some(block) = stream.next().await {
        if let MessageContentBlock::Text(text) = block? {
            print!("{text}");
            stdout().flush()?;
        }
    }
    println!();
    Ok(())
}

The reply depends on the model and the sampling seed. examples/local_model.rs is the runnable version: it prints download progress on stderr, prints Thinking blocks, and prints loaded on <device>. It reads the variables listed under Environment variables.

candle selects its quantized SIMD kernels when it compiles, so build for the host CPU. A generic build is several times slower.

export RUSTFLAGS="-C target-cpu=native"

$env:RUSTFLAGS = "-C target-cpu=native"

cargo run --release --example local_model --features provider-local

To run another preset, set CUCA_LOCAL_MODEL first:

export CUCA_LOCAL_MODEL="qwen3-0.6b-q8_0"

$env:CUCA_LOCAL_MODEL = "qwen3-0.6b-q8_0"

PowerShell keeps $env: values for the rest of the session; Remove-Item Env:CUCA_LOCAL_MODEL clears one.