Gaussian Mixture Attention: Linear-Time Sequence Mixing via Probabilistic Latent Routing

In three linesGaussian Mixture Attention (GMA) replaces standard attention with probabilistic routing through K learned Gaussian mixture components. Queries and keys map to responsibility vectors in a shared latent space. GMA avoids explicit N×N matrix materialization, reducing memory complexity to O(NK) instead of O(N²). Competitive on long-context classification, but behind SDPA and Mamba on WikiText-103.

Summary generated by Claude — human-verified