Reproduce “TherA: Thermal-Aware Visual-Language Prompting for Controllable RGB-to-Thermal Infrared Translation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dong-Guw Lee, Tai Hyoung Rhee, Hyunsoo Jang, Young-Sik Shin, Ukcheol Shin, Ayoung Kim
TherA introduces a framework for controllable RGB-to-Thermal Infrared (TIR) translation, leveraging a Thermal-Aware Vision-Language Model (VLM) and a VLM-conditioned diffusion model to generate diverse and thermally plausible pseudo-TIR images. It surpasses existing methods in fidelity and generalization, and enables modulation of thermal appearance via natural language or reference images, significantly alleviating TIR data scarcity.
Key insights
- Incorporating a physically grounded, structured thermal context through a VLM (TherA-VLM) is crucial for generating realistic and plausible RGB-to-TIR translations, moving beyond simple pixel-level style transfer.
- The use of rich, latent thermal embeddings from TherA-VLM, rather than generic text or raw RGB, provides superior conditioning for diffusion models, leading to significant performance gains in thermal image synthesis.
- Controllable thermal image generation is achievable by manipulating VLM inputs, enabling the simulation of diverse thermal scenarios, addressing issues like thermal inversion, and providing flexibility for downstream applications.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.