Reproduce “ScribbleEdit: Synthetic Data for Image Editing with Scribbles and Text” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Anya Ji, George Ma, Téa Wright, Yiming Zhang, David M. Chan, Alane Suhr, Somayeh Sojoudi
Researchers at the University of California, Berkeley, developed ScribbleEdit, a synthetic dataset designed to train image editing models to interpret both freehand scribbles for spatial guidance and natural language for semantic details. Fine-tuning models on this dataset significantly enhances their ability to generate spatially accurate and semantically consistent image edits, outperforming baselines on metrics like Histogram Intersection and RMSE.
Key insights
- Fine-tuning models on a specialized dataset like ScribbleEdit substantially improves their capacity to jointly interpret and act upon multimodal instructions for image editing.
- Off-the-shelf, pretrained models often struggle with abstract scribble inputs, frequently producing artifacts or failing to execute intended edits, highlighting the need for targeted training data.
- The incorporation of image conditioning, alongside scribbles and text, aids models like ControlNet in preserving background structures and enhancing the realism of generated edits.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.