Open sourcing InfiniteKV: a KV cache that files old tokens as 104-byte searchable records in RAM or on disk instead of deleting them. Mistral-7B answered from token 76,747, 2.3x past its trained window. Colab demo

In three linesInfiniteKV compresses KV cache into 104-byte searchable records stored in RAM or disk instead of deleting old tokens. Mistral-7B correctly answers at token 76,747 (2.3× its 32,768 training window). One million tokens requires ~3 GB instead of 122 GB.

Summary generated by Claude — human-verified