vLLM

In one line

Dispatches unified requests to a vLLM server's OpenAI-compatible chat completions route.

You need

The provider-vllm feature and a running vLLM server. An API key is optional.

Read this if

You route requests through ProviderEndpoint::Vllm.

This adapter runs the code on the OpenAI page unchanged. Its own differences:

• The default base URL is http://127.0.0.1:8000/v1.
• The API key is optional. A key is sent as a bearer token only when one is set.

Everything else on this page points to the matching section of the OpenAI page.

Endpoint

The default base URL carries /v1. Any base URL you set is used as given, so it must carry /v1 too when the server serves under it. The adapter appends only /chat/completions and trims a trailing / first.

Authentication

• Authorization: Bearer <api_key> is sent when the client has an API key.
• With no key, no auth header is sent and dispatch does not fail.

Request translation

Same as OpenAI request translation. The model is the request's model.

Response stream

Same as OpenAI response stream. reasoning_content deltas become Thinking blocks.

Token usage

Same as OpenAI token usage. The translator keeps the last usage object it sees, so a server that sends usage on every frame yields the final total.

Thinking

Same as OpenAI thinking: req.thinking maps to reasoning_effort.

Errors

The provider name in a frame error is OpenAi, because the translator is shared. A malformed frame or an error frame ends the stream after the error.

Limits

Same as OpenAI limits. There is no adapter cap and no request timeout.

Example

The smallest streaming turn: the Vllm variant against a local server, with no API key. The model id is the one the server reports under GET {base}/models.

cargo add cuca --path /path/to/cuca --features provider-vllm
cargo add tokio --features rt,macros
cargo add tokio-stream

export CUCA_BASE_URL="http://127.0.0.1:8000/v1"
export CUCA_MODEL="google/gemma-4-e4b"

$env:CUCA_BASE_URL = "http://127.0.0.1:8000/v1"
$env:CUCA_MODEL = "google/gemma-4-e4b"

use std::io::{Write, stdout};

use cuca::types::{MessageContentBlock, ProviderEndpoint};
use cuca::{CucaClient, UnifiedRequest};
use tokio_stream::StreamExt;

#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let base_url =
        std::env::var("CUCA_BASE_URL").unwrap_or_else(|_| "http://127.0.0.1:8000/v1".to_string());
    let model = std::env::var("CUCA_MODEL").unwrap_or_else(|_| "google/gemma-4-e4b".to_string());

    let client = CucaClient::builder()
        .with_provider(ProviderEndpoint::Vllm)
        .with_base_url(base_url)
        .build()?;

    let request = UnifiedRequest::new(model)
        .add_system_message("You are concise.")
        .add_user_message("Say hello.")
        .set_max_tokens(128);
    let mut stream = client.generate_stream(request).await?;

    let mut text_blocks = 0usize;
    let mut thinking_blocks = 0usize;
    while let Some(block) = stream.next().await {
        match block? {
            MessageContentBlock::Text(text) => {
                print!("{text}");
                stdout().flush()?;
                text_blocks += 1;
            }
            MessageContentBlock::Thinking { .. } => thinking_blocks += 1,
            _ => {}
        }
    }
    println!("\nblocks: {text_blocks} text, {thinking_blocks} thinking");
    Ok(())
}

cargo run

The provider-vllm feature comes from the cargo add line above.