ToolSense: A Diagnostic Framework for Auditing Parametric Tool Knowledge in LLMs

In three linesToolSense is an open-source diagnostic framework to audit actual tool understanding in LLMs. Applied to ToolBench (~47k tools), it reveals a knowledge-retrieval dissociation: five parametric model configurations collapse by 50-64 percentage points on realistic ambiguous queries, falling below embedding baselines, despite strong performance on standard benchmarks.

Summary generated by Claude — human-verified