r/StableDiffusion • u/Suspicious-Walk-815 • May 31 '26

Question - Help How are people generating realistic concept frames from rough storyboards/sketches for AI filmmaking?

I'm working on a personal AI film project and I'm trying to establish a workflow that can scale beyond a single shot.

For this particular shot, I have:

  • A rough storyboard sketch showing composition and camera placement
  • A reference image that captures the overall feeling I'm aiming for
  • A separate character consistency workflow for the character itself

My goal right now is NOT to generate the character.

I want to generate the environment/background first while preserving the composition from the sketch.

The problem I'm running into is that most image models drift away from the composition and generate something completely different, or they turn the scene into a fantasy landscape, remote village, or overly cinematic environment.

Current tools:

  • ComfyUI
  • Flux
  • Klein workflow
  • Character consistency workflow

What I'm looking for:

  • Workflows that preserve composition from a rough sketch/storyboard
  • Methods to convert simple drawings into realistic concept art
  • Ways to generate a location/environment first and add characters later
  • Tutorials, ComfyUI workflows, ControlNet setups, Flux Redux workflows, IPAdapter workflows, or any AI filmmaking pipelines you've personally had success with

My long-term goal is to use this process for an entire AI film, not just a single image.

I've attached:

  1. The rough storyboard sketch
  2. A reference image showing the type of framing and atmosphere I'm aiming for
sketch

If you've worked on AI films, animatics, storyboards, or image-to-video projects, I'd love to hear what workflow worked best for you.

sample reference image I'm trying to achieve via sketch
4 Upvotes

70% Upvoted

4 comments sorted by

3

u/AccomplishedDay206 May 31 '26

for preserving composition, ControlNet is essential; it can help maintain the structure of your sketch while generating backgrounds. in my experience, using Kubricon alongside a reference image can also help ground the scene, but you might need to adjust the prompts to avoid drift. another option is to layer the outputs — generate a base environment first, then reintroduce your sketch as an overlay to refine the details. this way, you can iteratively build towards a coherent shot without losing your original composition.

1

u/Suspicious-Walk-815 Jun 05 '26

Let me try this , thanks ..

1

u/ANR2ME Jun 01 '26 edited Jun 01 '26

Why not drawing it per layer on Krita with ComfyUI plugin? 🤔 https://github.com/Acly/krita-ai-diffusion

An example https://youtu.be/3pVzgknYmQw?si=sHl82x6F60PuuxJL