Reproduce “Structured Extraction from Business Process Diagrams Using Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pritam Deka, Barry Devereux
Researchers at Queen's University Belfast developed a VLM-based pipeline to extract structured JSON data from Business Process Model and Notation (BPMN) diagrams presented as images. Their approach, leveraging prompt engineering and optionally OCR, achieved high F1 scores with top-tier models and revealed that a hybrid DFS+BFS prompting strategy significantly improved extraction of relational elements.
Key insights
- Top-tier proprietary VLMs (e.g., GPT-4o, GPT-4.1) demonstrate strong inherent visual and semantic understanding for BPMN extraction, often showing minimal or negative gains from additional OCR input.
- Mid-tier VLMs benefit substantially from OCR integration, indicating that external textual cues improve recall and F1 scores when the VLM's native text grounding is less robust.
- Structurally guided prompt engineering, particularly the DFS+BFS hybrid, significantly improves VLM performance for complex symbolic diagrams by enforcing logical traversal and capturing relational dependencies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.