Reproduce “From Scope to Script: An Automated Report Generation Model for Gastrointestinal Endoscopy” on OpenResearch

About this paper

Evandros Kaklamanos, Kristjana Kristinsdottir, Jonathan Huang, Dustin Carlson, Rajesh Keswani, John Pandolfino, Mozziyar Etemadi

Researchers developed a transformer-based model with a two-stage training framework to automatically generate detailed gastrointestinal endoscopy reports from procedural images. This system produced reports demonstrating concordance with physician-documented findings, with pre-training on image-level captions proving crucial for performance.

Key insights

• The initial image-level caption pre-training stage is critical for learning generalized vision-language features, with its absence leading to a 72.96% reduction in ROUGE score, highlighting the necessity of this two-stage approach.
• Cross-attention maps reveal that the model grounds its textual outputs to specific visual features in the endoscopic images, localizing findings like polyps to the correct regions, which provides interpretability.
• Static images limit the model's ability to infer precise spatial measurements or procedural actions (e.g., biopsies) not explicitly visible in captured frames, indicating a need for more dynamic or context-rich inputs in future work.

Summary by alphaXiv.