Reproduce “Conditional End-to-End Audio Transforms” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Albert Haque, Michelle Guo, Prateek Verma

This work presents a fully-differentiable, end-to-end sequence-to-sequence model for audio transformations that eliminates complex hand-crafted pipelines by learning style and content representations directly from spectrograms. It achieves the highest Mean Opinion Scores and lowest Mel-Cepstral Distortion compared to baselines for both voice conversion and musical style transfer, effectively handling novel vocabulary and pitches.

Key insights

  • The model autonomously learns meaningful latent representations of acoustic attributes, successfully clustering sounds by pitch classes and discerning harmonic overlaps without explicit supervision.
  • Optimal acoustic context size for transformation quality varies across audio domains; speech benefits from an intermediate 50ms window, while musical notes are less sensitive to immediate context variations.
  • A unified end-to-end architecture can generalize effectively across distinct audio transformation tasks, such as voice conversion and musical instrument timbre transfer, while handling novel content.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.