Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Caches

In three linesSAECache introduces a semantic-aware eviction policy for LLM prefix caches. Not all tokens are equally worth caching: different token types (system prompts, user queries, tool outputs, reasoning) show up to 756x variation in reuse rates. SAECache uses a multi-queue architecture with online learning to adapt priorities, achieving 1.4x-2.7x TTFT improvement over production baselines.

Summary generated by Claude — human-verified