Reproduce “A Hormone-inspired Emotion Layer for Transformer language models (HELT)” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Eslam Reda, Sara El-Metwally
A framework named Hormone-Inspired Emotion Layer for Transformer Language Models (HELT) augments T5 with a biologically-inspired emotion layer to enable emotional text generation. The system predicts continuous hormone-like values to dynamically modulate internal representations, achieving 85.5% accuracy in hormone prediction and being preferred by human evaluators in 77% of test cases over a baseline.
Key insights
- A continuous, multi-dimensional representation of emotions, inspired by the human endocrine system, provides an effective inductive bias for modeling complex emotional states in language models.
- Transferring linguistic comprehension by initializing hormone attention Key/Value projections from pre-trained T5 weights significantly improves prediction accuracy and training stability.
- The use of orthogonal query initialization and auxiliary diversity and margin losses is crucial for encouraging distinct, interpretable attention patterns and confident prediction of extreme hormone activation values.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.