Reproduce “SAFT: Structure-Aware Fine-Tuning of LLMs for AMR-to-Text Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rafiq Kamel, Filippo Guerranti, Simon Geisler, Stephan Günnemann
The paper "SAFT: STRUCTURE-AWARE FINE-TUNING OF LLMS FOR AMR-TO-TEXT GENERATION" by researchers at the Technical University of Munich introduces a fine-tuning framework that integrates graph-structured information into Large Language Models without architectural changes. It achieves new state-of-the-art results for AMR-to-text generation on the AMR 3.0 dataset, improving BLEU scores by up to 3.5 points over previous baselines and showing larger gains on more complex and document-level AMRs.
Key insights
- Graph-structured information can be effectively incorporated into LLMs in an architecture-agnostic manner by additively injecting precomputed, projected positional encodings derived from the magnetic Laplacian.
- The magnetic Laplacian's eigenvectors provide a robust, direction-aware representation of graph topology that significantly enhances an LLM's ability to reason over complex semantic structures like AMRs.
- Explicit structural inductive bias leads to greater performance gains on more structurally complex inputs (e.g., deeper AMRs, document-level AMRs) compared to traditional fine-tuning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.