Reproduce “Loss-Guided Auxiliary Agents for Overcoming Mode Collapse in GFlowNets” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Idriss Malek, Aya Laajil, Abhijith Sharma, Eric Moulines, Salem Lahlou
LGGFN trains an auxiliary GFlowNet to seek trajectories where the main model has high training loss, while the main model retains its original reward and learns from samples from both models. On a modified balanced-bit-sequence task at length 40, it generated 7,746 distinct complete valid sequences among 16,000, versus 102 for SAGFN, with exploration error of 0.02 versus 0.98. This result is specific to that setting.
Key insights
- Leveraging the main model's training loss provides a direct and computationally efficient signal for 'poor understanding,' enabling highly targeted exploration unlike indirect novelty signals.
- The approach helps prevent partial-support stationary points by ensuring non-zero gradient mass in regions where the model is performing poorly, accelerating learning.
- An implicit curriculum is created as the main model's loss naturally decreases during training, causing the auxiliary agent to progressively focus on harder then easier exploration targets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.