MicroSpec: Accelerating Speculative Decoding with Lightweight In-Context Vocabularies

In three linesMicroSpec reduces active vocabulary by 40x (under 3k tokens) during speculative decoding without additional training. The technique exploits temporal locality in language generation and integrates asynchronous GPU memory management. End-to-end speedup of 1.12-1.32x vs EAGLE-2.

Summary generated by Claude — human-verified