Reproduce “Upsampling artifacts in neural audio synthesis” on OpenResearch

About this paper

Jordi Pons, Santiago Pascual, Giulio Cengarle, Joan Serrà

Pons et al. systematically investigate the origins and characteristics of upsampling artifacts in neural audio synthesis models, which typically use layers like transposed convolutions to convert low-dimensional representations into high-resolution audio. The study characterizes the distinct artifacts produced by common upsampling layers and explores how training and architectural choices influence their presence, demonstrating that while transposed and subpixel convolutions achieve high objective scores, nearest neighbor interpolation offers better perceptual quality by avoiding tonal artifacts.

Key insights

• Different upsampling layers exhibit distinct intrinsic artifacts: transposed convolutions and subpixel convolutions generate 'tonal artifacts,' while interpolation layers introduce 'filtering artifacts' due to fixed frequency responses.
• Spectral replicas, especially those from DC offsets created by biases and non-linearities, propagate and amplify artifacts throughout stacked upsampling layers.
• Although training can effectively compensate for initial artifacts in transposed and subpixel convolutions to achieve high objective scores, interpolation-based methods like nearest neighbor offer superior perceptual quality by inherently avoiding tonal artifacts.

Summary by alphaXiv.