Reproduce “Image2Reverb: Cross-Modal Reverb Impulse Response Synthesis” on OpenResearch

About this paper

Nikhil Singh, Jeff Mentch, Jerry Ng, Matthew Beveridge, Iddo Drori

Image2Reverb generates room impulse-response spectrograms from a single image, allowing dry audio to be acoustically transformed to resemble the depicted environment without directly measuring the space. A depth-augmented, Places365-initialized conditional GAN achieved a mean T60 error of −6.03%, substantially outperforming a visual nearest-neighbor baseline, while expert listeners found generated and measured reverberation broadly comparable for several indoor scene categories.

Key insights

• Scene-recognition pretraining was the strongest contributor to generalization: removing it increased mean T60 error from −6.03% to 43.15% and raised the standard deviation from 78.8% to 144.3%.
• Depth information and the explicit T60 objective provided smaller but consistent improvements, and controlled depth inputs shifted predicted mean T60 from 2.01 to 3.62 seconds, indicating limited acoustic steerability.
• The model learned to attend to visually meaningful regions such as walls, ceilings, windows, and wall–ceiling boundaries, although single images cannot determine hidden geometry, source–receiver position, materials, or directional acoustics.

Summary by alphaXiv.