Tokenization in Transformers v5: Simpler, Clearer, and More Modular

In three linesTransformers v5 simplifies the tokenization API with a more modular and clearer architecture. Changes aim to reduce complexity and improve code maintainability while preserving backward compatibility.

Summary generated by Claude — human-verified