RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways

In three linesRoVE proposes a parameter-free modification to RoPE (Rotary Position Embeddings) making values position-sensitive through simultaneous rotation with keys. GPT-2 models (124M, 354M) show empirical gains on few-shot in-context learning, out-of-distribution perplexity, and long-context retrieval tasks.

Summary generated by Claude — human-verified