LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding

In three linesLazyAttention is an attention mechanism optimizing key-value caching for RAG and in-context learning by deferring positional encoding. It enables zero-copy KV reuse at arbitrary positions, reducing time-to-first-token by 1.37× and increasing throughput by 1.40× versus Block-Attention.

Summary generated by Claude — human-verified