Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality

In three linesWebGraphMix selects pretraining data by analyzing Common Crawl web graph topology. The method computes centrality scores without model training or labeled data, then mixes central and peripheral documents. At 400M–1B parameters, 1:1 ratio achieves 41.4% average (+1.6pp vs uniform sampling), 43.8% combined with quality scores.

Summary generated by Claude — human-verified