Back to feed
Reddit r/MachineLearning·

My offline ablation said -0.19pp. The production retrain said +1.11pp. [D]

Signal
72
Hype
15
In three linesML engineer reports offline ablations (retrain with/without feature) contradicted production results. Four changes: Best Offer feature (+0.12pp offline → -0.19pp prod), auction data backfill (+0.37pp prod), outlier trimming (-0.19pp offline → +1.11pp prod), CatBoost encoder. Root causes: train/serve skew, unmeasured distribution shift, training population drift, baseline instability.
Read source
Your take?
EvalsBenchmarks

Summary generated by Claude — human-verified