Rational Sparse Autoencoder

In three linesSparse autoencoders (SAEs) for mechanistic interpretability rely on fixed activations (ReLU, JumpReLU, TopK). This paper introduces Rational Sparse Autoencoder (RSAE), replacing the fixed encoder activation with a trainable rational function. RSAE improves reconstruction and sparsity trade-offs across three open-weight language models while maintaining feature-level interpretability.

Summary generated by Claude — human-verified