Reproduce “Let's Transfer Transformations of Shared Semantic Representations” on OpenResearch

About this paper

Nam Vo, Lu Jiang, James Hays

A method is introduced for learning semantic transformation functions in a data-rich source domain (e.g., 2D synthetic images or text) and transferring them to a data-scarce target domain (e.g., 3D renders or natural images) through a shared semantic embedding space. This approach achieves comparable retrieval performance to direct target-domain training for image-to-image transformations and outperforms baselines for text-to-image transformations, especially for novel concepts.

Key insights

• Semantic transformation logic can be effectively learned in a data-rich source domain and successfully transferred to a data-scarce target domain by establishing a robust shared semantic embedding space.
• Learning an explicit transformation function within the shared embedding space provides superior performance compared to simpler methods, such as embedding arithmetic or multi-stage domain translation, for complex semantic changes.
• The approach demonstrates enhanced robustness in handling queries involving novel or out-of-vocabulary concepts in the target domain by performing transformations directly within the shared embedding space, rather than relying on potentially unreliable domain-specific translations.

Summary by alphaXiv.