Reproduce “APEX: Probing Neural Networks via Activation Perturbation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tao Ren, Xiaoyu Luo, Qiongxiu Li
APEX (Activation Perturbation for EXploration) introduces an inference-time framework that directly probes neural network internal representations by perturbing hidden activations. This method reveals two distinct regimes: a small-noise regime exposing local, sample-dependent semantic structures and a large-noise regime uncovering global, model-level biases, notably identifying backdoored models through a "target-aligned collapse" in predictions.
Key insights
- APEX offers a unified structural perspective, establishing input perturbation as a constrained special case within its framework, thereby accessing representation properties previously inaccessible.
- The approach reveals two empirically distinct regimes: a small-noise regime for local, instance-specific structure and a large-noise regime for global, model-level stationary behavior.
- The robustness of APEX's findings across different noise distributions, perturbed layers, and activation functions suggests the observed phenomena are general properties of activation perturbation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.