ai reading/researchpaper clm llm
Core Idea
Context Language Models (CLMs) let a model edit its working context instead of keeping an ever-growing conversation history.The paper argues that this helps long-running agents stay accurate, use less compute, and learn how to manage context.
A Context Language Model (CLM) is not a separate model architecture like a new Transformer or a new foundation model. It is a way of using an existing language model with a different context-management setup.
The Problem
An LLM normally appends each turn to the context window: Turn 1 ⟶ Turn 2 ⟶ Turn 3 …
After a long task, the context can contain the original request, search results, failed attempts, terminal and tool output, old plans, corrections, summaries, and current work.
Some of that information is useful. Much of it becomes irrelevant.
Traditional systems use an external harness to decide when to summarize or delete information, retrieve memories, or compact the conversation. (3. Context and Subagents > 2. Compaction forgetting well) Other systems let the model choose from a predefined set of context-management actions.
The authors argue that both approaches limit the model to strategies designed by people. They ask why the model should not manage its context itself.
Proposed Solution
CLMs let the model manage its own context directly. Rather than keep an append-only history or rely on an external system to summarize and delete information, a CLM treats its live context as an editable file.
The model can use standard file and shell operations to remove irrelevant content, summarize long sections, update notes, preserve important facts, or reorganize its state. Edits are synchronized with the model’s active context.
- Traditional LM:
old context + new output → larger context - CLM:
old context → model edits/reorganizes it → better next context
How a CLM context file differs
The key difference is what the file represents and whether it is the model’s active context.
An agent-managed external memory file (for example, Memory.md or AGENTS.md) is separate storage. The model may write notes there and retrieve them later, but the file is not automatically part of its current context. A system must decide what to load.
A CLM context file is the model’s live context. Editing it changes what the model sees in its next inference step. The paper contrasts this with systems that store information elsewhere and retrieve it into an otherwise append-only context.
| External memory file | CLM context file |
|---|---|
| Separate from active context | Is the active context |
| Requires retrieval/loading | Edits are synchronized automatically |
| Usually stores long-term/archive info | Stores current working state |
| Agent/harness decides what to retrieve | Model directly rewrites its working context |
| Context can still keep growing | Model can delete/replace/reorganize context in place |
Reinforcement Learning (RL) in CLMs
The Bitter Lesson and Richard Sutton
- CLM provides the capability: the model can directly edit its own live context.
- RL improves the policy: it teaches the model when to edit, what to keep, what to remove, and when to summarize.
- The training reward considers:
- Task success/correctness
- Efficiency, such as using less context-related compute
- The paper uses a success-gated efficiency advantage: it rewards efficiency among successful trajectories. This avoids rewarding a model for deleting useful information and failing.
- After RL, context-management behavior can become part of the model’s learned weights, rather than relying only on prompt instructions.
CLM gives the model the ability to manage context.
RL trains it to manage context well and use less compute.
Context Edit Cost and Caching
Prefix Caching Deep Dive, Prefix vs KV vs Prompt Caching
When a CLM edits something in the middle of its context, standard prefix caching becomes less efficient. The cache can only be reused up to the first changed token, so the unchanged text after that edit usually has to be re-prefilled/recomputed, increasing inference cost. The paper measures this using prefix-reuse FLOPs.
To reduce this overhead, the paper proposes Suffix Cache Reuse (SCR). Instead of recomputing the entire suffix after an edit, SCR reuses cached states for unchanged surviving tokens and computes mainly the newly changed content.
In their experiment, SCR reduced compute from about 10.98 to 7.14 PFLOPs per question while keeping the same task accuracy, roughly a 35% compute reduction.
Context editing improves memory management, but can break normal prefix-cache reuse; SCR is introduced to recover much of that lost efficiency.
What If an AI Could Edit Its Own Memory?