Reproduce “Benchmarking LLMs on File System Design and Implementation” on OpenResearch

About this paper

Yuqi Xue, Daixuan Li, Jian Huang

Researchers at the University of Illinois Urbana-Champaign developed -Bench, a benchmarking framework comprising 505 tasks, to evaluate Large Language Models on file system design and implementation. The study found LLMs excel at basic knowledge and implementation tasks (up to 95.8% pass rate) but struggle with complex tasks like debugging and performance optimization (as low as 37.6%), while detailing specific failure types and the varying efficacy of mitigation strategies.

Key insights

• LLMs demonstrate strong basic understanding and coding abilities but struggle to integrate these for complex reasoning-heavy tasks required in sophisticated file system development.
• Failures are primarily driven by logical inference errors, incomplete reasoning, and suboptimal coding implementations, rather than simple instruction following or basic knowledge recall.
• Output instability is a notable characteristic, particularly for less capable models, indicating a higher propensity for 'guessing' when faced with uncertainty.
• Prompt engineering techniques like knowledge self-review and error feedback can improve performance, but Retrieval-Augmented Generation (RAG) did not provide significant benefits and sometimes negatively impacted results for frontier models.

Summary by alphaXiv.