Reproduce “Stay CALM: Calibrated Online Aggregation for Multi-Horizon Forecasting Under Delayed, Overlapping Feedback” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shivesh Prakash
CALM adjusts an online forecasting aggregator’s learning rates for correlated feedback from overlapping future targets, without changing the underlying forecasting models. It scales each rate by an estimated loss variance and an overlap factor, using horizon as a default. Across 90 pre-registered runs, its test MSE was lower than Proceed’s in 88, but calibrated BOA won most often.
Key insights
- Delayed feedback and overlapping feedback are distinct issues: overlapping targets make successive losses dependent, so standard online learning rates can respond too strongly to correlated noise.
- In the paper’s analysis, comparable learning behavior across horizons requires reducing the learning rate approximately in proportion to the overlap factor; simulations support scaling updates by the horizon or an estimated overlap factor.
- Calibration improved robustness rather than guaranteeing the best average ranking: CALM was not usually the top method, but its worst-case error stayed within 3.5% of the best method in every reported run.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.