Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Models

In three linesEmpirical study comparing 11 tokenizers across 11 Southeast Asian languages. Standard BPE tokenizers structurally favor high-resource languages and Latin scripts. Parity-aware BPE achieves best efficiency-equity trade-off; Morphology-Driven Byte Encoding delivers superior semantic performance but at higher computational cost.

Summary generated by Claude — human-verified