Reproduce “SCP-NL2TL: Selective Conformal Prediction with Semantic Verification for Natural Language to Temporal Logic Specifications” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yixuan Wang, Licheng Luo, Yu Fu, Kaidi Xu, Yue Dong, Mingyu Cai

SCP-NL2TL presents a selective translation framework that equips black-box Natural Language to Temporal Logic (NL2TL) systems with an accept-or-abstain decision layer, statistically guaranteeing a user-defined bound on the rate of accepted incorrect specifications. The framework achieved consistent risk control across various logic languages and improved discrimination of errors with a fusion score achieving 0.706-0.920 AUROC, while an Out-of-Distribution (OOD) filter enhanced robustness to distribution shifts.

Key insights

  • Fusing back-translation fidelity and self-consistency provides complementary signals, significantly improving the discrimination between correct and incorrect temporal logic specifications, achieving AUROC values up to 0.920.
  • Conformal Risk Control (CRC) effectively controls the joint risk of accepting erroneous specifications, providing distribution-free statistical guarantees that reliably adhere to user-specified error budgets for individual outputs.
  • An instruction-level out-of-distribution (OOD) filter serves as a crucial upstream safeguard, maintaining the robustness of risk guarantees under distribution shifts by proactively deferring atypical inputs before translation, also offering computational benefits.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.