Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models

In three linesStudy on UTF-8 reliability in byte-level models (355M params, 80B multilingual tokens). UTF-8 validity converges 2× slower than perplexity (4.2B vs 2.1B tokens). Rare characters generate more valid UTF-8 than frequent ones, revealing over-specialization of common character representations.

Summary generated by Claude — human-verified