Improving Parquet Dedupe on Hugging Face Hub

In three linesHugging Face improves Parquet deduplication on its Hub. The optimization reduces data redundancy and speeds up dataset loading. Technical details on the algorithm and performance impact included.

Summary generated by Claude — human-verified