Reproduce “OmniNet: A unified architecture for multi-modal multi-task learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Subhojeet Pramanik, Priyanka Agrawal, Aman Hussain
OmniNet presents a unified architecture, extending the Transformer with a spatio-temporal cache and gating mechanism, to enable multi-modal and asynchronous multi-task learning. The model achieved competitive performance across diverse tasks while reducing total parameters by three-fold compared to individual task-specific models and demonstrated zero-shot learning capabilities on novel video-based tasks.
Key insights
- A truly unified architecture, extending the Transformer, can efficiently process multi-modal inputs and perform asynchronous multi-task learning through shared representations.
- Explicitly modeling spatio-temporal dependencies with a dedicated cache and gated attention mechanism is crucial for handling diverse real-world data like images, text, and video.
- Multi-task training can significantly reduce model parameters and enhance generalization, allowing for effective zero-shot application to novel, related tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.