Reproduce “DeSRPA: Decoupled Speech Role-Playing Agent via Inference-Time Intervention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenqiu Tang, Zhen Wan, Takahiro Komamizu, Ichiro Ide
The DeSRPA framework introduces a decoupled approach for Speech Role-Playing Agents that adapts character personas and acoustic expressions without fine-tuning, leveraging inference-time intervention on frozen Large Language Model (LLM) and Text-to-Speech (TTS) backbones. This method achieves an Emotion Execution Accuracy of 0.701 and Speaker Similarity of 0.886, demonstrating robust consistency between persona and vocal expression while ranking comparably to proprietary systems in multimodal evaluations.
Key insights
- Inference-time intervention with subtle control vectors can effectively steer both LLM-based persona generation and TTS-based acoustic expression without requiring extensive model fine-tuning.
- Decoupling cognitive reasoning from acoustic rendering preserves the LLM's intrinsic capabilities while allowing for precise, synchronized control over character-specific textual and vocal nuances.
- The use of SAEs for cognitive vectors and style subtraction for acoustic vectors provides an efficient and interpretable method to generate targeted, orthogonal control signals for persona and emotion.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.