Decompose Sparsely Where You Should, Absorb Densely Where You Should No

In three linesSparse autoencoders (SAEs) assume all activations are amenable to sparse decomposition. This work adds a low-rank dense bottleneck in parallel with standard SAEs to capture a causally important dense component. On Gemma-2-2B layer 12, a rank-24 bottleneck reduces dense latents by 84% while improving sparse probing performance.

Summary generated by Claude — human-verified