Reproduce “CALLIOPE: Private Insights into AI Use, Without Embeddings” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Edith Cohen, Vadym Doroshenko, Badih Ghazi, Charlie Harrison, Pritish Kamath, Alexander Knop, Ravi Kumar, Ethan Leeman and 6 more
CALLIOPE discovers conversation topics by summarizing documents, using differentially private keyword counts to propose labels, and assigning summaries with an LLM. On CLINC-OOS, human raters preferred CALLIOPE in 76% of comparisons at reported ε=0.1, tied 6%, and preferred URANIA 18%. This sampled preference is not an accuracy score, and the methods’ reported ε values represent different total privacy budgets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.