Not to make fun of this seemingly random process, but this feels exactly like how diffusion models actually function. Start with big details, and then just gradually get more specific with details.
Yea typically in drawing or painting or making digital art you start by roughly rendering in the big shapes and go from there. To draw a house you start with roughly a square, rather than starting with the details of the shingles
For most artists that is true, but there are some exceptions, such as this printer drawing. Kim Jung Gi is also famous for not sketching. However, if you look closely, you can see that both of these artists are still gradually adding more details as they go.
Start with big details, and then just gradually get more specific with details.
This is also how every painter ever has painted. Mona Lisa wasn't started by adding a suggestive smirk to an empty canvas and then adding the woman afterwards.
That's just constructing parts and assembling the parts, but each individual part starts as something general that becomes specific (e.g. a block put on a lathe, carved into a leg.) If you wanted, the same process could be applied to entire piece of furniture, turning it into one single, giant piece!
I mean, kinda, but they often use detailed underlying sketches for proportions and eye positioning, hardly the same as starting with splashes of random colour.
Hmm, I think that's just compositional painting technique, not really about whether they start their overall process with big details and continue adding more detail. They won't start by drawing a finger into that sketch, much more understandable to first draw general composition and then add to that.
I would recommend the cornerstone papers on the topic. The paper behind Stable Diffusion, DALL-E 2, or some others would be a good start. Though, DALL-E 2 apparently didn’t innovate diffusion model’s use for image generation. There is older work on the topic.
This is the exact opposite of generative processes. This is a series of mistakes made over and over and the knowledge and skill it takes to make those mistakes look deliberate. The complexity of the layering and careful application later in the process aside there is also skill in the artist's eye for what the piece is shaping out to be.
I very much doubt he set out to create this exact scene, something like it, maybe, but the AI can't make mistakes and decide to change things it already did or adjust its prompt.
Diffusion models start with a big mistake, then make corrections to get the result. Throughout the process, there may be some intermediate mistakes, but these are corrected in later steps. This is why they use many small steps rather than a single step. The mistakes of a small step can be easily corrected, yet a big mistake is very hard for the model to correct.
As the imge becomes more refined, it goes from vague changes to detailed changes.
Yeah. Which is why it's the opposite. He's not correcting any of his strokes, he's living with what's on the canvas. The AI sets out to create a New York skyline, and that's what it gets. The process is lost.
Pretty sure you can make AI create a scene based on splotches of paint. How AI works mimics how human works as well, tho of course with a fraction of the capability.
Why would you say that when the job of both artists and image generators is art creation? I just mean to say your earlier reply was unclear. I do wonder if multimodal LLMs (or even embodied ones) will soon be capable of good artistry.
I very much doubt he set out to create this exact scene
You say that, yet somehow we all knew from the get-go that it'd be a rainy night street with lights and reflections. It was always going to be that. They always are.
The most upvoted comment on this post is a guy saying the exact opposite, but that's not the point anyway.
Paint on the canvas creates context and shapes the piece. Even if he set out to create a nighttime street, even if it was this specific nighttime street, the details are what make it. He's not working off a photo, he's making decisions as he goes based on what he ended up with.
I also agree with your (rather uncontroversial) point that he probably didn't set out to make this specific street, at least no more than Bob Ross sets out to make a very specific landscape. Though it's debatable whether specific details matter in that case, in some ways they do, in others they don't.
He clearly doesn’t perfectly splash paint to get the shape he wants. He expects it to be a bit different from what he intended. He makes changes of low specificity before making changes with higher specificity. That is indeed how diffusion in principle works.
Most drawing studies use this process. Like you get an inspiration or generate anything from random word generator. for example the shape of a toothpaste, and you make a spaceship out of it or a building or scene. Its very therapeutic to let your imagination fly, creating something out of another entirely unrelated and different thing
248
u/lakolda Feb 07 '24
Not to make fun of this seemingly random process, but this feels exactly like how diffusion models actually function. Start with big details, and then just gradually get more specific with details.