Reproduce “Learning Keypoints for Robotic Cloth Manipulation using Synthetic Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Thomas Lips, Victor-Louis De Gusseme, Francis wyffels
Researchers at Ghent University developed a synthetic data pipeline for training keypoint detectors to identify semantic locations on "almost-flattened" cloth items. This method, when combined with real-world fine-tuning, increased mean Average Precision to 74.2% and reduced the Average Keypoint Distance to 8.7 pixels, nearly halving the error compared to training on real data alone.
Key insights
- Combining synthetic data pre-training with real-world fine-tuning proves highly effective, significantly boosting keypoint detector performance on cloth manipulation tasks compared to training on real data alone.
- Prioritizing diversity in synthetic data generation, such as using procedural single-layer meshes and random asset library textures, can yield better sim-to-real transfer than striving for high visual fidelity with limited asset variety.
- A quantifiable reality gap of approximately 20 percentage points in mAP and 11 pixels in AKD remains between purely synthetic-trained models and real-world performance, highlighting areas for future simulation and asset realism improvements.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.