The Illusion of Intervention: Your LLM-Simulated Experiment is an Observational Study

In three linesLLMs simulating users in intervention experiments produce biased observational studies. Trained on observational data, they induce implicit population drift across treatment conditions, distorting effect estimates. Authors propose negative control outcomes to diagnose bias and persona specification adjustments to mitigate drift.

Summary generated by Claude — human-verified