Reproduce “StrucText-Eval: Evaluating Large Language Model's Reasoning Ability in Structure-Rich Text” on OpenResearch

About this paper

Zhouhong Gu, Haoning Ye, Xingzhou Chen, Zeyang Zhou, Hongwei Feng, Yanghua Xiao

Researchers at Fudan University developed StrucText-Eval, an automatically generated benchmark covering 8 diverse structure-rich text formats and 29 tasks, to evaluate Large Language Models' (LLMs) reasoning abilities. Experiments revealed that current state-of-the-art LLMs possess a superficial understanding of structural information, showing a significant performance degradation on complex tasks and a substantial gap compared to human performance.

Key insights

• LLMs exhibit superficial structural understanding, heavily biased by training data distribution; performance varies significantly across different structured languages, with JSON showing higher accuracy due to its prevalence.
• The 'w/ Hint' prompting strategy generally improved performance on standard tasks by providing human-engineered guidance, but its impact was limited on highly complex tasks.
• There is a substantial human-LLM performance gap in structural reasoning, with human participants achieving 92.6% accuracy on complex tasks while state-of-the-art LLMs struggled to reach 70%.

Summary by alphaXiv.