Reproduce “TaskLAMA: Probing the Complex Task Understanding of Language Models” on OpenResearch

About this paper

Quan Yuan, Mehran Kazemi, Xin Xu, Isaac Noble, Vaiva Imbrasaite, Deepak Ramachandran

Google Research developed TaskLAMA, a high-quality human-annotated dataset and robust evaluation metrics, to probe the capabilities of large language models (LLMs) in Structured Complex Task Decomposition (SCTD). The study found LLMs significantly outperform baselines in generating task steps and adapting to context, but struggle with explicit pairwise temporal dependency prediction, a weakness mitigated by soft-prompt tuning.

Key insights

• Large Language Models possess extensive intrinsic knowledge for generating relevant task steps, demonstrating a substantial performance advantage over traditional decomposition baselines.
• LLMs exhibit strong context understanding, significantly improving the quality and specificity of generated task steps when provided with explicit contextual assumptions.
• There is a clear distinction in LLM capabilities: they are proficient at generating steps in a sensible linear sequence (implicit ordering) but struggle with explicit pairwise temporal dependency reasoning without targeted tuning.

Summary by alphaXiv.