Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression

In three linesSoftmax-attention transformers can implement preconditioned Richardson iteration for in-context Gaussian kernel regression. Authors construct a single-head transformer with O(log(1/ε)) blocks achieving ε-accurate prediction on prompts of length N, where softmax attention produces a Gaussian-kernel operator and ReLU MLP layers perform local scalar arithmetic.

Summary generated by Claude — human-verified