Reproduce “What Breaks Under Pruning in Smart Homes, and When? Evaluating LLM Degradation Across Architectures and Task Complexity” on OpenResearch

About this paper

Congjing Zhang, Vashishtha Patil, Henning Lange, Usman Aleem

An evaluation of structured pruning in smart-home LLM agents shows that compression affects environment grounding before general action intent. Dense models exhibit sharp accuracy declines at high pruning levels and increasingly refuse valid commands, whereas expert pruning in a sparse MoE model removes 70% of experts with only a reported 0.37-point accuracy change, though aggregate scores can conceal substantial instance-level prediction changes.

Key insights

• Pruning weakens contextual grounding earlier than general action structure: device and value selection deteriorate more than operation and argument prediction, especially at aggressive dense-pruning levels.
• Task difficulty alone does not determine pruning sensitivity; partially executable requests are particularly fragile because they require selective execution and rejection, while simple requests remain vulnerable to device ambiguity.
• Aggressive pruning changes the execution policy itself. For Qwen3-4B at 50% pruning, false refusals on executable requests rise from 7.7% unpruned to 56.2% with Angular and 80.8% with FLAP.

Summary by alphaXiv.