Do Value Vectors in Deep Layers Need Context from the Residual Stream?

In three linesResearchers propose Bank of Values (BoV), replacing context-dependent value vectors with context-free vectors stored as sparse parameters in the last third of layers. On 135M and 780M models, BoV improves validation loss and performance across 21 benchmarks with reduced compute and memory.

Summary generated by Claude — human-verified