Reproduce “From Prompts to Trees: Effective LLM-Guided Tree Generation for Few-Shot Tabular Classification” on OpenResearch

About this paper

Yue Qiu, Zekang Du, Yiqun Diao, Bingsheng He, Qinbin Li

LLMT addresses few-shot tabular classification by asking an LLM to generate feature rules, then arranging them into a decision tree using a small labeled sample. This separates rule generation from data-guided tree construction. Across eight datasets with two examples per class, it averages 0.753 accuracy versus 0.701 for the strongest baseline in the table, but does not win every dataset.

Key insights

• Feature metadata appears useful: masking names and task descriptions reduced average accuracy by 20.14 percentage points on six newer datasets, though this does not rule out pretraining memorization.
• Under matched budgets of atomic conditions, individual rules generally covered correct predictions better than longer conjunctive reasoning paths.
• Separating LLM rule generation from data-driven tree assembly allows semantic priors to inform an interpretable model without requiring LLM inference at deployment.

Summary by alphaXiv.