Context Memorization for Efficient Long Context Generation

In three linesTraining-free method to optimize long-context inference: attention-state memory externalizes prefix into lightweight lookup-based memory of precomputed attention states. On LLaMA-3.1-8B, improves in-context learning at 1K-8K tokens, reduces attention latency by 1.36x at 8K, outperforms full-attention RAG with 20% less memory.

Summary generated by Claude — human-verified