Reproduce “CLIPascene: Scene Sketching with Different Types and Levels of Abstraction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yael Vinker, Yuval Alaluf, Daniel Cohen-Or, Ariel Shamir
Researchers from Tel Aviv University and Reichman University developed CLIPascene, a method for generating abstract vector sketches of complex scenes with disentangled control over fidelity and simplicity. The approach produces a matrix of sketches ranging from geometrically precise to semantically loose and from detailed to sparse, maintaining high semantic recognition and generating editable vector output.
Key insights
- Utilizing activations from different intermediate layers of a ViT-based CLIP model provides an effective mechanism to control the fidelity of abstraction, moving from precise geometry to semantic emphasis.
- Implicitly learning stroke removal probabilities through a dedicated network (MLPsimp) and a sparsity-ratio loss (Lratio) allows for perceptually smoother and more semantically coherent simplification than explicit stroke count methods.
- Decomposing complex scenes into foreground and background elements is critical for stable, artifact-free, and controllable abstraction results.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.