Are Sparse Autoencoder Benchmarks Reliable?

In three linesCritical audit of SAEBench, the de-facto standard evaluation suite for sparse autoencoders (SAEs). TPP and SCR metrics fail multiple reliability tests and should not be used. Other metrics show higher reseed noise and lower discriminability than assumed. Only sae-probes demonstrates acceptable reliability, but struggles to distinguish architecture variants.

Summary generated by Claude — human-verified