Reproduce “Evaluating Shrinking (Experience Report)” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alperen Keles, George Miao, Leonidas Lampropoulos
An empirical comparison of shrinking in QuickCheck, Hedgehog, and Falsify separates the cost of simplifying an existing failure from the cost of finding one. Across four Haskell workloads and several generator designs, QuickCheck is usually faster, while integrated shrinkers preserve valid generator structure more reliably but do not consistently produce smaller or closer-to-minimal counterexamples.
Key insights
- Generator construction often matters more than the shrinking framework: rejection-heavy type-based generators perform worse than generators that directly preserve validity or build inputs through the system API.
- Integrated shrinking retains failing candidates more frequently, especially Hedgehog, suggesting stronger preservation of generator invariants; however, this advantage does not reliably translate into faster shrinking or better final counterexamples.
- Shrinking outcomes depend on candidate ordering, internal representations, workload invariants, and initial input size, so no framework is uniformly best across effectiveness and efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.