An expressivity analysis of hierarchical modelling in deep transformers via bounded-depth grammars

In three linesTheoretical analysis of deep transformer expressiveness through bounded-depth, non-recursive context-free grammars. Authors explicitly construct transformers with positional attention whose depth scales linearly with grammar depth, demonstrating these architectures can encode abstract grammatical states into linearly separable subspaces within the residual stream.

Summary generated by Claude — human-verified