Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention

In three linesTwo complementary mechanisms improve transformer attention: Energy-Gated Attention (EGA) selects informative tokens via linear projection; Morlet Positional Encoding (MoPE) replaces sinusoidal encodings with learned Gaussian wavelets. On TinyShakespeare, their combination achieves +0.119 validation loss improvement, exceeding the sum of individual parts.

Summary generated by Claude — human-verified