Reproduce “MortarBench: Evaluating Mortgage Loan Origination Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Matthew Toles, Yunan Lu, Manav Munjal, Bojun Liu, Yuanhao Deng, Stephanie Selig, Derek Rindner, Cheng Li and 1 more
Researchers from Columbia University and Tidalwave introduce MortarBench, the first public benchmark for evaluating LLM agents in mortgage loan origination, featuring synthetic, internally consistent bank statements and loan application data. An agent augmented with a Confidence Reflection Inference for Transactions (CRIT) framework improved F1 accuracy by up to 8.6% and highlighted systematic biases in LLM decisions regarding transaction origin language and informal value transfer systems.
Key insights
- State-of-the-art LLMs demonstrate limited accuracy (max F1 80.8%) on mortgage underwriting tasks, particularly struggling with Transaction ID questions and exhibiting oversensitivity through a high false positive rate.
- A significant portion of LLM errors stems from consistent false beliefs and reasoning failures (e.g., transaction misclassification, lack of domain knowledge) rather than purely stochastic variation.
- LLM agents exhibit systematic biases in classifying transactions based on the language of origin of names (especially non-Latin scripts) and the use of informal value transfer systems, raising ethical concerns.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.