AI Tool That Converts Images to Sketches

Check out CLIPasso – the AI tool that generates sketches based on images. The tool performs CLIP-guided Semantically-Aware Object Sketching that allows both varying levels of abstraction and preserving key visual features.

The devs point out that "even with a very minimal representation [the tool] can recognize both the semantics and the structure of the subject depicted".

This object sketching method achieves different levels of abstraction which is guided by geometric and semantic simplifications. The devs harness CLIP's (Contrastive-Language-Image-Pretraining) ability to extract identical semantic concepts in sketches and images. The tool controls the abstraction degree by varying the number of strokes. So, the produced sketches demonstrate different levels of abstraction still managing to retain recognizability including structure, and key visual components of the initial subject.

To generate the sketch, the tool begins with extracting the salient regions of the input image that allows it to define the initial locations of the strokes. Going through the following steps of optimization, the tool feeds the stroke parameters to a differentiable rasterizer which produces the rasterized sketch. Finally, both the resulting sketch and the original image are fed into CLIP to define a CLIP-based perceptual loss.

The developers note that using the intermediate layers of a pre-trained CLIP model is a crucial point of this method that allows it to constrain the geometry of the output sketch.

You can learn more about CLIPasso here. Also, don't forget to join our new Reddit page, our new Telegram channel, follow us on Instagram and Twitter, where we are sharing breakdowns, the latest news, awesome artworks, and more.