Reproduce “Weighted Automata Extraction from Recurrent Neural Networks via Regression on State Spaces” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Takamasa Okudono, Masaki Waga, Taro Sekiyama, Ichiro Hasuo
Researchers at the National Institute of Informatics developed a method to extract Weighted Finite Automata (WFAs) from real-output Recurrent Neural Networks (RNNs) by applying regression on the RNN's continuous state space. This approach yielded WFAs that were highly accurate in approximating RNN behavior, even for non-regular language tasks, and achieved approximately 1,300 times faster inference compared to the original RNNs.
Key insights
- Using regression on the RNN's internal state space allows for effective abstraction of continuous states, enabling the application of L*-style learning to extract WFAs from quantitative RNNs.
- The method can extract meaningful and interpretable WFA approximations even from RNNs modeling behaviors that exceed WFA expressivity, such as bounded versions of non-regular languages like balanced parentheses.
- Adjusting the search space exploration (e.g., through a concentration threshold `M`) based on training data characteristics significantly improves accuracy, especially for RNNs trained on non-uniform datasets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.