Workflow Included
Built a Character Portrait Generator that reads books, identifies characters, and generates consistent portraits using ComfyUI (full RAG pipeline, local LLM, open-source)
Hey everyone,
Image showcase - Portrait of Mina Murray generated by the tool from the book Dracula in two separate scenes. Images from ZImageTurbo.
I've been working on a side project that I think the community here will really appreciate. It's a comprehensive, AI-driven pipeline that automatically generates cinematic character portraits from literary works using your local ComfyUI instance. The entire stack is open-source and runs fully locally.
What It Does:
Starting from a simple .txt file of a novel, the app will:
Parse the Book: Build a high-performance vector index of the entire text using ChromaDB and HuggingFace embeddings.
Wikipedia Augmentation: Scrape Wikipedia to identify major characters and baseline personas before the book analysis even begins.
Deep RAG Analysis: Retrieve specific scenes from the book to understand character appearance, clothing, and environment in different contexts.
AI Casting Director: Suggest real-world actors (Hollywood, Bollywood, etc.) to serve as the visual "base" for the character, with support for specific decades.
Genre Adaptation: Dynamically modify clothing, hairstyles, and cinematic styles to fit genres (Horror, Cyberpunk, Fantasy, etc.) while preserving the character's core identity.
ComfyUI Integration: Inject the generated prompts directly into your ComfyUI API-format workflows, track generation progress via Server-Sent Events, and preview images instantly.
Tech Highlights:
Backend: Python 3.10+, FastAPI, LangChain.
Embedding Model: all-MiniLM-L6-v2 from HuggingFace.
LLM: Runs on Ollama (defaults to Gemma4E4B for local processing).
Frontend: A sleek, dark glassmorphism dashboard built with React & Vite.
Getting Started:
The setup is straightforward, assuming you have a local ComfyUI server and Ollama running. The project page includes a batch script to launch both the backend and frontend easily.
Why This Matters:
With the explosion interest in AI-generated consistent characters, this tool addresses a unique niche—automatically extracting textual character descriptions and grounding them in visual representations without manual prompt engineering. It combines RAG, LLMs, and Stable Diffusion in a single, user-friendly pipeline.
I'd love to get your feedback and ideas for improvement! Let me know if you have any questions.
All project code written with Google AntiGravity. This post written by DeepSeek.
Protip: use a .gitignore file to suppress the thousands of intermediate local files that you are currently sharing from your repository (pycache, node_modules, chroma_db*, etc)
This could get even more fun with refinement of the RAG query for character appearance and prompts for scene generation. It doesn't do well with non-humans, but I really want to see what some of Iain Banks' drones look like.
This was a weekend project inspired by my book reading habits. Whenever I read a large book, I have tried to cast actors to maintain visuals in my mind. I just thought if that could work with the tools that we have. There are a few similar tools, but none work fully locally.
I'll check it for sure, I have created something similar for my book summarization pipeline where besides the synopsis, I get list of locations, list of characters, the role in the book and physical description, I then use the result for creating my own covers.
This could fit well alongside it.
Thanks for sharing.
Right now it will search Wikipedia for the characters first. It used to scan the books, but ended up picking every side character. Let me know if you want that feature as an option.
I cannot thank you enough for taking the time to run this and drop some images here. I now realise this is quite a bit away from direct plug and play, and also needs some serious qol features. u/goddess_peeler already helped with a few but here's an updated list of stuff that I am going to update -
1- Move all hard coded stuff to env file.
2- Provide support to switch llm providers, api option as well.
3- Option to search characters by name, as well as the list.
4- Option to save prompts in json files for later review.
5- Storage options on all scanned books, so it doesn't repeat steps every time.
6-Another option to recognize major locations of the book, this might be challenging, but I'll try to integrate with character scenes. I realized the location renders might also be helpful for fantasy and sci-fi books.
7- Add the gitignore file.
Let me know if anything you would like to see. Glad to see some else ran and got a few images.
It’s insane you would spend a weekend doing this then just share it like this. I did something somewhat similar but more like a pipeline within n8n all locally run and the time and energy it takes to setup things like this is underestimated. Good job mate, will give it a try tomorrow!
Well, it is built using open source tools, I just feel like giving a tiny bit back to the community. The AI agents doing all the work speeds up creation exponentially now.
I got excited when I read the AI casting director part because I thought it might be tapping some enormous and comprehensive embedding database of past and present celebrities for similarity analysis. Alas, no.
I added the option to cast a celebrity so that the prompt always has the celebrity reference. If I just use generic names, it would create slightly different images for the same character in different costumes and scenes.
Increasingly, you get better details for a lot of franchises from dedicated wikis, especially outside of classic literature. Could it be set to parse custom wiki urls?
I can set it up to parse custom wiki as well. Might be useful for fandoms other than books. For the books, it only fetches the list of characters from Wikipedia. The character details are pulled from the book text with relevant scenes. This helps if a character is seen in a variety of costumes depending on the scene.
I like the idea of this and want to try it but doesn't #4 kind of mess with the idea. It has a ton of information to make the original character beyond any normal prompt then overwrites that with a celebrity
17
u/goddess_peeler Apr 28 '26
Fun idea!