Constructing Evaluation Datasets for Procedural Reasoning: Balancing Naturalness, Grounding, and Multi-Hop Coverage

In three linesStudy on generating evaluation datasets for procedural reasoning. Three strategies compared: strict generation from TMK (Task-Method-Knowledge) models, transcript-first generation with post-hoc TMK filtering, and TMK-aware generation. Across 690 QA pairs and 23 instructional topics, strict TMK generation achieves 96.5% grounded questions and 92.6% usable items.

Summary generated by Claude — human-verified