Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Models

In three linesTheoretical study on geometric limits of feature representation in transformers. Authors establish a framework based on linear representation and superposition hypotheses, showing representational capacity depends on vectors-to-dimensions ratio (k/d) rather than raw count. Analysis of dozens of open-source models reveals two classes based on orthogonality constraint ε.

Summary generated by Claude — human-verified