Memory

In one line

MemoryPlugin compacts the request's message list when it crosses a size trigger, and renders an in-memory working graph into requests as context.

You need

The plugin-memory feature.

Read this if

You are registering MemoryPlugin, tuning compaction, or working with the MemoryGraph.

Feature and entry type

Supporting types:

• Compaction: CompactionStrategy, CompressionAction, CompressionReport, Budget.
• Usage reporting: ContextUsage, ContextUsageObserver, ContextWindowResolver.
• Extension seams: Summarizer, VectorStore.
• Graph: GraphContextConfig, GraphNode, GraphRelationship, GraphDirection, MergePolicy, MergeReport, GraphSnapshot, GraphImportReport.

Hooks

MemoryPlugin acts on on_request only. on_stream_chunk and on_response_complete use the trait defaults.

On every request, on_request runs these steps in order:

1. Count the request's tokens and hand a ContextUsage reading to every ContextUsageObserver. An observer error fails the request.
2. When warn_fraction is set and usage reaches it, append one system message that starts with CUCA context warning:. The marker makes the warning one-shot.
3. Resolve the Budget. When the request is over it, run the compaction pipeline.
4. When graph_context is set, render the working graph into a system message right after the first system message.

Triggers

• max_messages, max_tokens and max_fraction select the trigger. max_messages wins when combined with either token budget.
• max_tokens and max_fraction are mutually exclusive; setting both is rejected at construction. The default max_fraction is Some(0.8), so a config that sets max_tokens must also set max_fraction: None.
• max_messages triggers when the list is longer than the value. max_tokens triggers when usage reaches it. max_fraction multiplies the resolved context window.
• The window comes from ContextWindowResolver when one is set and returns a value, otherwise from context_window_tokens.
• With no trigger set, the budget is Budget::Unlimited and on_request compacts nothing.

Compaction pipeline

• Strategies run in the order of strategies. A strategy whose extension seam is absent no-ops, and a strategy error is recorded and the pipeline continues.
• With a trigger, the pipeline re-measures after each acting strategy and stops once under budget. It runs at most 4 passes and stops early when a pass makes no progress.
• The most recent user message and the first system message survive every strategy.
• Every strategy that removes messages also removes the ToolResult message of each dropped ToolCall, so no result outlives its call.
• MemoryPlugin::compress(&mut messages) is the out-of-band entry point. It runs the whole strategies list once with no budget and returns a CompressionReport.
• MemoryPlugin::count_tokens() approximates: it sums tiktoken-rs counts of "<role> <text>" per message, over Text blocks only, and models no per-message framing tokens.

Working graph

• The plugin owns one MemoryGraph behind a mutex: nodes with labels and properties, and directed weighted relationships.
• merge_graph(other, policy), replace_graph(graph), replace_snapshot(snapshot), snapshot() and graph() access it. Merging happens under one lock hold.
• The rendered message starts with the marker CUCA graph memory:. A later request replaces it in place, and it is removed once the graph is empty.
• The graph step runs after compaction, so compaction cannot delete the graph message, and its token cost is not counted against the request that triggered compaction.

Configuration

MemoryConfig implements Default; MemoryPlugin::new(config) validates it.

• encoder_name is a tiktoken-rs encoder name or a model name such as "gpt-4o".
• context_window_tokens is the fallback window, and the base for max_fraction and warn_fraction.
• offload_turns is how many turns DropTurns removes per pass.
• GraphContextConfig defaults: max_nodes 64, max_relationships 128.

Default strategies, in order: Offload { turns: 10 }, Summarize { turns: 10 }, DeduplicateFileReads { tool_name: "read_file" }, ClearToolResults { keep_pairs: 3 }, ClampOversizedMessages { max_part_tokens: 4096 }, SlidingWindow { keep_messages: 40 }, DropObservations, DropTurns.

MemoryPlugin::with_extensions(config, summarizer, store) attaches the Summarizer and VectorStore that Summarize and Offload need.

Caps and at-cap policy

Message list:

Working graph:

warn_fraction injects the one-shot near-limit warning before the bound is reached.

Errors

Every error MemoryPlugin itself raises is PluginError::Internal.

• MemoryPlugin::new and with_extensions fail when both max_tokens and max_fraction are set, or when the encoder cannot load.
• count_tokens, compress, graph, merge_graph, replace_graph, snapshot and replace_snapshot fail when the encoder or graph mutex is poisoned.
• replace_snapshot fails on an invalid snapshot: a duplicate node or relationship id, a non-finite weight, or an endpoint missing from the nodes. The working graph is untouched.
• MemoryGraph::add_relationship fails when from or to names an unknown node.
• on_request returns the token-count failure and any error a ContextUsageObserver returns. Either fails the request before dispatch.
• A strategy error never fails the request. It lands in CompressionReport::last_error and the pipeline continues. A failed Offload returns the turns to their original positions.

Example

examples/memory.rs runs four turns with max_messages: Some(6), a 256 token window, warn_fraction: Some(0.2) and a single SlidingWindow { keep_messages: 4 } strategy.

let memory = Arc::new(MemoryPlugin::new(MemoryConfig {
    context_window_tokens: 256,
    max_messages: Some(6),
    warn_fraction: Some(0.2),
    observers: vec![gauge as Arc<dyn ContextUsageObserver>],
    strategies: vec![CompactionStrategy::SlidingWindow { keep_messages: 4 }],
    ..Default::default()
})?);
let client = CucaClient::builder()
    .with_provider(ProviderEndpoint::LlamaCpp)
    .with_base_url(base_url)
    .register_plugin(memory as Arc<dyn CucaPlugin>)
    .build()?;

Turn 4 carries both edits. The gauge crosses warn_fraction, so one warning message is appended, and the nine message prompt exceeds the six message trigger, so SlidingWindow trims it to four. The surviving shape is SAUS because the first system message and the most recent user message are never removed. The reply Unknown. follows from the dropped turns: reaching them again is the offload seam of the vector store.

cargo run --example memory --features "provider-llamacpp plugin-memory"

Trigger: max_messages 6, window 256 tokens, warn at 20%
Strategy: SlidingWindow { keep_messages: 4 }
Prompt shape letters: S system, U user, A assistant

Turn 1: "The vault is in Lisbon. Reply with: noted"
  gauge: 16/256 tokens (6%), window resolved false
  prompt: 2 messages in, 2 out (SU), warning false
  reply: noted

Turn 2: "The key rotates on friday. Reply with: noted"
  gauge: 29/256 tokens (11%), window resolved false
  prompt: 4 messages in, 4 out (SUAU), warning false
  reply: noted

Turn 3: "The owner is ops@example.com. Reply with: noted"
  gauge: 43/256 tokens (17%), window resolved false
  prompt: 6 messages in, 6 out (SUAUAU), warning false
  reply: noted

Turn 4: "Where is the vault? Answer in one word."
  gauge: 56/256 tokens (22%), window resolved false
  prompt: 8 messages in, 4 out (SAUS), warning true, compacted
  reply: Unknown.

The replies and token counts depend on the model; the message counts do not. The output above came from google/gemma-4-12b-qat.

Related pages

• Plugins: hook order, error handling and the plugin tier rules.
• Memory discipline: why every growing structure carries a cap.
• Entity extraction: produces graph deltas applied through merge_graph or replace_graph.
• Vector store: implements the VectorStore seam that Offload writes to.