Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements

In three linesIshigaki-IDS-Bench is a benchmark for evaluating generation of Information Delivery Specification (IDS) XML files from BIM requirements. On 166 expert-validated examples in English/Japanese, the 10 best LLMs reach 65.6% macro F1 for content agreement, but only 27.7% pass the IDS Content audit. Models struggle to generate XML conforming to IDS standards and IFC vocabulary constraints.

Summary generated by Claude — human-verified