Reproduce “Generalizing Trust: Weak-to-Strong Trustworthiness in Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Martin Pawelczyk, Lillian Sun, Zhenting Qi, Aounon Kumar, Himabindu Lakkaraju

Investigating weak-to-strong trustworthiness in language models, Harvard University researchers found that fairness and robustness properties can transfer and even be amplified in stronger models through explicit regularization during both weak model training and weak-to-strong transfer. Privacy, however, did not similarly generalize, indicating unique challenges for its transfer.

Key insights

  • Weak-to-strong trustworthiness generalization is feasible for fairness and robustness (OOD and adversarial), but does not naturally occur for privacy.
  • Explicit trustworthiness regularization applied during both weak model training and the weak-to-strong transfer process (Weak+WTS TFT) is crucial for effective and enhanced trustworthiness transfer.
  • Privacy transfer poses a distinct challenge in the weak-to-strong setting, as larger models tend to memorize more data, limiting direct generalization of privacy guarantees from a weaker model.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.