Reproduce “Domain-Specific Fine-Tuning and Prompt-Based Learning: A Comparative Study for developing Natural Language-Based BIM Information Retrieval Systems” on OpenResearch

About this paper

Han Gao, Timo Hartmann, Botao Zhong, Kai Lia, Hanbin Luo

This research systematically compares domain-specific fine-tuning and prompt-based learning for natural language-based BIM information retrieval, revealing that a hybrid approach combining fine-tuned intent recognition with advanced LLMs for question answering achieves 90.80% accuracy. The study identifies significant scalability limitations of current LLMs when processing large-scale BIM data.

Key insights

• A hybrid approach, combining fine-tuned XLNet for intent recognition with prompt-based GPT-4o for table QA, provides the most robust performance for NLI in BIM environments.
• Domain-specific fine-tuning is exceptionally effective for intent recognition, achieving 99.14% accuracy, outperforming prompt-based LLMs due to specialized training data.
• Advanced prompt-based LLMs (GPT-4o) demonstrate superior latent reasoning capabilities for complex tabular data questions, achieving 91.95% accuracy in QA, but their performance is sensitive to model versions and input scale.
• Current LLM-based approaches face significant scalability limitations when handling large-scale BIM models due to context window constraints and quadratic complexity, leading to performance degradation on extensive tabular inputs.

Summary by alphaXiv.