Large-scale Near-deduplication Behind BigCode

In three linesBigCode built large-scale near-deduplication infrastructure to clean code data. The system identifies and removes near-duplicates across billions of files, improving training dataset quality for code models.

Summary generated by Claude — human-verified