Reproduce “Engine-Transfer-Bench: An Evidence-Based Benchmark for Document Compilation Engine Selection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Prajwal S. Venkateshmurthy
Engine-Transfer-Bench (ETB) establishes an evidence-based benchmark for evaluating document compilation engines, providing a standardized corpus, tasks, metrics, and protocols to characterize engine reliability, latency, and text consistency across multiple operating systems. The work empirically demonstrates significant performance and consistency differences among engines and introduces ETB-Porta, a companion system that leverages these insights for engine recommendation and portability gating in CI/CD pipelines.
Key insights
- TeX Live engine reliability issues are concentrated in a small, engine-specific minority of documents (13.2% of the corpus), rather than being uniformly distributed, suggesting a mixture model for overall success rates.
- Significant and systematic latency differences exist, with Tectonic consistently compiling 2.23 to 3.24 times faster than classical TeX Live engines on multi-engine portable documents due to architectural distinctions.
- Successful compilation by multiple engines does not guarantee semantic identity; 25.5% of portable documents show notable text-level divergence ($S_{pdf} < 0.95$) in their generated PDFs, even after all engines succeed.
- Dominant compilation failures in TeX Live engines are architectural (e.g., font-stack mismatches, multi-file layout issues) rather than simple missing packages, while Typst failures primarily relate to missing assets in its distribution packaging.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.