Reproduce “Seven simple steps for log analysis in AI systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Magda Dubois, Ekin Zorer, Maia Hamin, Joe Skinner, Alexandra Souly, Jerome Wynne, Harry Coppock, Lucas Sato and 12 more
The authors present a seven-step framework for turning AI logs into structured evidence about system behavior or evaluation. It links questions to signals, develops and validates scanners, then analyzes results with human oversight. In a refusal-classification example, Sonnet 4 is credited with F1 0.998 and 99.0% accuracy across four labels on 400 samples; GPT-5 is also named as scanner model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.