Reproduce “Skill-Based Few-Shot Selection for In-Context Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shengnan An, Bo Zhou, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Weizhu Chen, Jian-Guang Lou
SKILL-KNN improves few-shot example retrieval by asking a frozen language model to describe the operations required by each query, then matching these skill descriptions rather than raw wording. Across semantic parsing benchmarks, the training-free approach generally outperforms standard embedding retrieval and performs competitively with fine-tuned selectors, reaching 78.3% execution accuracy on Spider and 82.7% with GPT-4 using its consistency variant.
Key insights
- Representing an example by the operations it requires can be more useful for retrieval than representing its surface vocabulary, especially under database and language shifts.
- A small number of manually annotated demonstrations is sufficient to guide skill extraction, and performance remains competitive even when the annotation set or operation coverage is reduced.
- The two aggregation strategies address different failure modes: consistency helps suppress diffuse random variation, whereas distinctiveness can exploit an informative rewrite despite occasional outliers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.