Reproduce “Waste-Bench: A Comprehensive Benchmark for Evaluating VLLMs in Cluttered Environments” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Muhammad Ali, Salman Khan
Researchers from Mohamed Bin Zayed University of Artificial Intelligence developed Waste-Bench, a comprehensive benchmark to evaluate Vision Large Language Models in cluttered waste classification environments featuring deformed objects. The evaluation revealed that current VLLMs exhibit inferior performance, with the best model achieving 57.52% accuracy, substantially below the human benchmark of 81.20%.
Key insights
- Current VLLMs demonstrate "inferior performance" in complex, cluttered waste classification tasks, indicating a substantial capability gap compared to human judgment.
- Models struggle with specific challenges including accurate counting of irregular objects, identifying rare or deformed items, distinguishing colors under occlusion, and differentiating similar materials in cluttered scenes.
- Performance significantly degrades when VLLMs are exposed to common visual distortions like noise, shading, and varied lighting, highlighting a lack of real-world robustness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.