Reproduce “IFHierBench: Hierarchical Instruction Following for Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuetian Mao, Chunyang Chen

Researchers from the Technical University of Munich developed IFHierBench, a novel benchmark to assess large language models' adherence to hierarchical output constraints in real-world applications. Evaluations show a marked decrease in prompt-level accuracy for all tested models as instruction nesting depth increases, with top models like GPT-5.5 reaching 53.7% and Claude Opus 4.6 42.8% overall, highlighting a significant capability gap in structured instruction following.

Key insights

  • Large language models universally exhibit a substantial decrease in prompt-level accuracy as the hierarchical depth of instructions increases, with performance notably degrading by 37-53 percentage points from flat to single-level nested constraints.
  • Analysis revealed distinct failure modes, including "content slippage" where individual constraints pass but overall adherence fails, and critical "structural failures" where models fail to generate the correct outer container elements.
  • Precise numeric constraints, such as exact character or word counts, posed the greatest challenge for LLMs, suggesting a specific area for targeted improvement in instruction following.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.