Reproduce “Embedding-perturbed Exploration Preference Optimization for Flow Models” on OpenResearch

About this paper

Sujie Hu, Chubin Chen, Jiashu Zhu, Jiahong Wu, Xiangxiang Chu, Xiu Li

E²PO (Embedding-perturbed Exploration Preference Optimization) fine-tunes generative flow models by injecting learned, structured perturbations into the text embedding space. This approach sustains discriminative variance and broadens exploration, leading to superior human preference alignment, increased diversity, and improved compositional reasoning compared to state-of-the-art methods.

Key insights

• The collapse of intra-group discriminative variance is a critical failure mode in RL-based optimization for generative models, directly causing learning signal loss and reward hacking.
• Leveraging structured, learned semantic perturbations provides a more robust and effective exploration mechanism than relying solely on unstructured stochastic noise.
• Dynamically balancing diverse semantic guidance during early generation phases with precise, unperturbed conditioning in later stages is crucial for achieving both high fidelity and diversity.

Summary by alphaXiv.