llama.cpp has a clever trick for speeding up KV cache decode

In three linesllama.cpp features a KV cache optimization that re-sends generated tokens to cache instead of waiting for next prompt, improving responsiveness. User reports latency reduction from 5-30s to near-instant on Qwen 3.6-35B with RX 7900 XTX (~100 tps).

Summary generated by Claude — human-verified