Reproduce “Towards Fair Pay and Equal Work: Imposing View Time Limits in Crowdsourced Image Classification” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gordon Lim, Stefan Larson, Yu Huang, Kevin Leach
Researchers from the University of Michigan and Vanderbilt University explored the effects of imposing view time limits in crowdsourced image classification tasks. The study found that a 1000ms view time limit maintained individual accuracy comparable to untimed tasks, demonstrated the robustness of consensus algorithms in preserving data quality under time pressure, and indicated worker preference for shorter limits based on psychometric scores.
Key insights
- An optimal view time limit (e.g., 1000ms for image classification) can achieve individual accuracy comparable to untimed tasks, balancing efficiency and performance.
- Extended viewing time for incorrect classifications often shows diminishing returns, suggesting that limiting view time can prevent unproductive effort without significantly compromising overall accuracy.
- Consensus algorithms are highly effective at maintaining and improving data quality, even when individual workers operate under time pressure, making time limits a viable approach for reliable data collection.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.