Reproduce “Ambient @ EgoProactive 2026 : Proactive Egocentric Assistance with Visually Grounded Supervision” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Logesh Kumar Umapathi
Ambient frames proactive egocentric assistance as a binary intervention-timing task rather than requiring a model to decide whether to speak while generating an utterance. A single-token verbalizer, dialogue history, and visually grounded synthetic labels improve balanced intervention performance, yielding 0.7179 macro-F1 for the 4.54B challenge submission and first place in the large division; vocabulary pruning produces a 1.9977B model with identical predictions on 544/544 evaluation decisions.
Key insights
- Separating the speak-or-silent decision from language generation substantially improves balanced performance: in-domain macro-F1 rises from 0.4517 to 0.7008 and G-mean from 0.4004 to 0.7007.
- Previous assistant decisions are essential for policy behavior; omitting dialogue history caused the model to repeatedly intervene, whereas the single-token formulation without fine-tuning collapsed to always-silent predictions.
- Visually grounded labels transfer better across domains than larger narration-only datasets: agent-generated supervision improves cross-domain G-mean despite being roughly ten times more expensive per clip.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.