Reproduce “Reversible Jump MCMC Simulated Annealing for Neural Networks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Christophe Andrieu, Nando de Freitas, Arnaud Doucet

The authors present a reversible-jump MCMC simulated-annealing method that jointly adjusts an RBF network’s centers and number of basis functions. Calibrated Bayesian priors make AIC, BIC, or MDL selection correspond to maximizing a joint posterior. On one robot-arm task, a 200-iteration MDL search selected 12 basis functions with test mean-square error 0.00512; this experiment does not establish performance across tasks.

Key insights

  • Classical model selection criteria (AIC, BIC, MDL) can be formally reinterpreted as maximizing a calibrated Bayesian posterior distribution.
  • Reversible jump MCMC, when integrated with simulated annealing, enables effective global search and simultaneous model selection for variable-dimension problems like RBF networks.
  • Analytically computing RBF amplitudes and linear coefficients (Rao-Blackwellisation) significantly enhances the algorithm's efficiency and reduces variance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.