Reproduce “Before You Continue: Evaluating Time-Sensitive Safety Awareness in Conversational Assistants” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Peidi Dong, Yunze Xiao, Yu-Lin Tsai, Yichen Zheng, Heyuan Huang, Weihao Xuan, Mark Dredze, Jen-tse Huang

TASC evaluates whether assistants apply time-dependent rules from earlier conversation turns to later requests. Across 13 models, they flagged 30.5% of violations in the main benchmark set. In a separate paired set, models missed 62.1% of violations despite correctly answering elapsed-time questions. These authored scenarios measure benchmark replies, not real-world incidents, and do not identify why the gap occurs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.