Reproduce “Doodle Your Keypoints: Sketch-Based Few-Shot Keypoint Detection” on OpenResearch

About this paper

Subhajit Maity, Ayan Kumar Bhunia, Subhadeep Koley, Pinaki Nath Chowdhury, Aneeshan Sain, Yi-Zhe Song

Researchers introduce a framework that enables few-shot keypoint detection on query photographs by leveraging a limited number of annotated sketches as support data, establishing a "source-free" cross-modal learning paradigm. The method consistently outperforms baseline approaches across various keypoint detection settings and demonstrates robust generalization to real free-hand sketches.

Key insights

• Human sketches can effectively serve as a "source-free" support modality for few-shot keypoint detection on photographs, offering a practical alternative when photo-based training data is scarce.
• Explicitly addressing the domain gap between sketches and photos at the keypoint embedding level, combined with learning style-invariant representations, is crucial for successful cross-modal transfer.
• The strategic use of auxiliary keypoints significantly augments data and enhances the framework's learning capabilities, leading to substantial performance improvements.

Summary by alphaXiv.