Grammatically-Guided Sparse Attention for Efficient and Interpretable Transformers

In three linesNovel sparse attention approach using grammatical roles (POS tags) to reduce quadratic complexity of Transformers. Two masking strategies tested on SST-2 with DistilBERT: hard mask (0.8200) and soft mask (0.8165) maintain full attention performance (0.8200) while reducing computational overhead.

Summary generated by Claude — human-verified