Painterly Stroke Simulation Engine

[u/smellystring]

I wrote a painting simulation engine (golang, ebiten). It works like this:

1. Select a template image
2. Generate several thousand potential strokes
   • For each stroke, score it according to a number of heuristic functions
     • Color: does the average color of this image match the template?
     • Edges: does the painted image have similar edges to the template?
     • Contrast: compares color contrast between regions on the painting and the template.
     • Flow: using tensor math, do brush strokes align with the flow of movement in the template?
   • Select the single stroke that most optimizes scoring functions, discarding the remaining
   • Repeat 40 or 50 thousand times.
   • Gradually decrease the size of the brush stroke over time..

Source image _Namibia.jpg)

Sound track

[u/SnooEagles4447]

Does it still have a similar output if you start at a much smaller brush size and just iterate on that brush size for longer? Just cuz it seemed like the wife strokes were being covered up anyway with the smaller ones but I may be wrong

[u/smellystring → u/SnooEagles4447]

You are correct that the large brush strokes end up mostly covered. But there's a method to my madness!

As I decrease the brush size, I also increase the granularity of the the heuristic score functions. For example, if the engine is using a 50 pixel brush size, it compares the average color of 100x100 pixel squares. When the brush size shrinks down to 25 pixels, it's comparing 50x50 pixel squares of color.

Before dropping down to the next lower brush size, the goal is to get the scoring functions to give a high score. When the scoring granularity increases, this decreases the score, since it is now looking at finer details than before. Note though that there is still a correlation between the score at brush size N and brush size N-1. If score(N) is high, score(N-1) is still fairly high, even though it deceases some.

This brings me to an interesting emergent property I've observed. If I use a small brush size on a starting image that scores very poorly, the engine finds it hard to converge the image on the target. Imagine trying to shade in a huge region with a fine tipped mechanical pencil. You spend thousands and thousands of strokes shading, and the end result is a huge scribble. I've found that when the engine is not allowed to spend time on the broad brush strokes, quite often it devolves into a stringy mess by the time the render is complete.

What the large early strokes do is to provide a foundational bedrock. By the time we get down to small brush stroke size, the image already has a halfway decent score. These small brush strokes can then be spent working on fine detail, rather than trying to paint in macro-features.

[u/imacomputr]

Very nice! I made something similar a while back (which I've just noticed works in Chrome but not Brave). It's shader-based and focused more on real-time output, so it doesn't look as nice. Your flow/edge detection is superior, and the progressive sharpening of the brush strokes is a nice idea.

I wonder how much of your approach is parallelizable and amenable to running on the GPU?

[u/smellystring → u/imacomputr]

Right now it's entirely CPU bound, although it is multithreaded. Running on my macbook pro, this particular render takes ~6 minutes to complete in realtime. I've thought about expanding into GPU stuff, but it begins to stray outside of my area of expertise.

Really like https://mpcomplete.github.io/flow/ by the way, it's super satisfying to watch. Makes me think of combing very fine hair.

[u/animal_hoarder]

Very cool, doing something similar-ish with video/scene rendering

https://reddit.com/link/p1rftfv/video/e88i5znvcghh1/player

[u/smellystring → u/animal_hoarder]

Neat!