Reproduce “PatRe: A Full-Stage Office Action and Rebuttal Generation Benchmark for Patent Examination” on OpenResearch

About this paper

Qiyao Wang, Xinyi Chen, Longze Chen, Hongbo Wang, Hamid Alinejad-Rokny, Yuan Lin, Min Yang

Researchers introduce PatRe, a new full-stage benchmark for patent examination that models the multi-turn interaction between examiners and applicants, involving both Office Action and rebuttal generation. Experiments reveal proprietary large language models generally perform better, and these models are more adept at reactive defense than proactive examination, often exhibiting a hyper-critical bias and logical inconsistencies in legal reasoning.

Key insights

• Proprietary Large Language Models consistently outperform open-source models, especially in the complex, adversarial reasoning demanded by rebuttal generation.
• LLMs exhibit greater proficiency in reactive defense (generating rebuttals) compared to proactive problem discovery (generating Office Actions), indicating an asymmetry in their legal reasoning capabilities.
• Current LLMs display a "hyper-critical bias," leading to high false-rejection rates and logical inconsistencies, often hallucinating prior art conflicts or over-extending legal statutes (e.g., §101, §102, §103, §112), which reflects an immaturity in balanced legal adjudication.
• Traditional lexical overlap metrics (e.g., ROUGE-L) poorly correlate with substantive legal validity in patent examination, underscoring the necessity of more sophisticated evaluation methods like the LLM-as-a-Judge paradigm.

Summary by alphaXiv.