Reproduce “A Reproducible and Realistic Evaluation of Partial Domain Adaptation Methods” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tiago Salvador, Kilian Fatras, Ioannis Mitliagkas, Adam Oberman
This paper presents a rigorous, reproducible evaluation of Partial Domain Adaptation (PDA) methods, revealing that many prior state-of-the-art results rely on unrealistic model selection using target labels. It demonstrates a performance drop of up to 30 percentage points without target labels and shows that just 100 randomly labeled target samples can restore near-optimal model selection performance.
Key insights
- Unsupervised model selection for PDA remains an open problem, with existing strategies largely failing to reliably select competitive models without target labels, often performing worse than a source-only baseline.
- A minimal budget of only 100 randomly labeled target samples (100-RND strategy) dramatically improves model selection, closely approximating the performance of an oracle with full target label access.
- Many PDA methods exhibit non-negligible instabilities across different random seeds, indicating that reported 'best' results might be due to lucky initialization rather than inherent robustness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.