A Policy-Driven Runtime Layer for Agentic LLM Serving

In three linesProposes intermediate runtime layer between agent framework and LLM serving engine. Introduces four primitives (observe, score, predict, act) to implement agent-aware policies (KV caching, batch shaping, speculation, fairness, safety). CacheSage, instantiated for cross-session caching, achieves +13 to +37 pp cache hit-rate lift, 12–29% lower TTFT, 6–14% higher throughput on five real multi-agent workloads.

Summary generated by Claude — human-verified