Reproduce “On the Performance of Concept Probing: The Influence of the Data (Extended Version)” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Manuel de Sousa Ribeiro, Afonso Leote, João Leite

Researchers at NOVA LINCS systematically investigate the influence of data characteristics on concept probing performance in Explainable AI. Their findings provide empirical guidelines, demonstrating that relevant concepts can be probed accurately with surprisingly few samples, data reuse does not degrade performance, and probes are moderately robust to label noise, which collectively enhance the practical applicability of concept-based interpretability methods.

Key insights

  • Original neural networks efficiently distill information about task-relevant concepts into their internal representations, making them easier to probe with less data.
  • Concept probing is scalable and effective even with very large original models, with larger models potentially providing slightly more beneficial information.
  • Reusing data from the original model's training set for concept probing does not negatively impact probe generalization.
  • While concept probes show moderate robustness to noise, real-world label noise can have a more pronounced impact than artificial random noise.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.