Reproduce “The Geometry of Ignorance: Detecting Suppressed Knowledge in LLMs through Activation Fragmentation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Adar Peleg, Dvir Alsheich, Tomer Ashuach, Avi Mendelson
The authors introduce an internal-representation method for screening whether language models retain information after unlearning. It compares hidden-state differences for concept-specific questions and identity-swapped versions, measuring whether those differences cohere across batches. Across tested models, the geometry distinguished known from fictional concepts; after unlearning, greater fragmentation correlated with lower recovery by five-shot in-context learning, not other attacks consistently.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.