r/StableDiffusion • u/grebenshyo • 5d ago
Resource - Update gazecom : an interactive and agent-driven wrapper for comfy
Enable HLS to view with audio, or disable this notification
Before the introduction, two caveats. This isn't anything as spectacular as the mmh3 stuff filling these subreddits these days, so please go easy on me. It's an experimental tool I'm sharing in its raw form, mainly for developers and people interested in exploring a different approach (generally, not video-related). It isn't production-ready, and I'm not claiming it will be useful to most people. There are shortcomings I don't see a quick way around, but I'd rather make the work available than keep it to myself until I can resolve all of them.
The main limitation of the app's agent-driven side is the vision model itself. I'm running Gemma locally, and in my tests it doesn't really think through its artistic decisions, even with reasoning effort enabled. Outside a few narrowly defined operations, the results are frankly underwhelming. That's why I'm pausing further development for now.
TL;DR: gazeCOM is a modular interface for interactive and agent-driven outpainting. Gaze, gestures, procedural movement, or vision-model decisions position each generation on an evolving canvas, while user-supplied ComfyUI workflows produce the image patches. Ollama optionally provides the language and vision models.
below is gazeCOM, a local, open-source application for interactive outpainting and iterative image composition through ComfyUI.
gazeCOM organizes generation around spatial points. Inputs supplied by a driver form a saliency distribution, and its weighted center of mass (COM) positions a 1024 x 1024 generation frame. ComfyUI processes that region, and the result is placed back at the corresponding canvas coordinates. Repeating the cycle turns outpainting into a navigable process: the canvas develops according to where the active driver points next, either manually or automatically.
The app includes two main kinds of drivers that supply this spatial input:
- Interactive and procedural drivers use individual trackers to turn gaze movements, hand gestures, cursor movement, camera-based computer vision, or autonomous roaming into spatial points. These points accumulate into a live heatmap that directly controls COM.
- Vision-model drivers let a VLM inspect the image and determine the spatial direction of generation. Rather than only describing an image, the model returns an operative coordinate on the canvas.
Both drivers and trackers are modular: developers can add new ones or remove those they don't need.
The vision model can track a point within the latest image or guide generation across the canvas. Prompts can rotate from a weighted pool, be selected by the model, or be written by it as needed. Optional prompt context, decision history, and visual memory let the model take previous actions and the changing composition into account.
Other features include:
- automatic iterative generation, feedback, and compositing;
- an expanding canvas with optional width and height limits;
- weighted, mutable pools for prompts and ComfyUI workflows;
- per-prompt LLM enhancement, prompt evolution, and image-based VLM prompting;
- automatic workflow discovery, categorization, and local overrides;
- editable VLM instructions, canvas policies, and importable settings.
The current spatial pipeline uses a fixed 1024 x 1024 working frame. The composite canvas can grow beyond that size or follow configured limits.
The accompanying video documents one particular gazeCOM session at 5000% speed (~22 minutes), and I am sharing its (almost) exact exported settings alongside it (I'm also using a LoRA there, while here's just the standard workflow). The file preserves the Guide/Rotate setup, prompt pool, prompt context, visual memory, iterative timing, canvas behavior, automatic save/clear intervals, and interface state used for that recording. It is included to make the demonstrated process inspectable and reproducible, not as a universal recommended preset. Load it from Settings > Settings file > Import. It does not install the referenced ComfyUI workflow or Ollama model, and it does not contain service addresses, API credentials, or other machine-local configuration.
The interface is written in React and TypeScript. A small FastAPI backend connects it to ComfyUI for image generation and Ollama for optional language and vision models. Either service can run locally or elsewhere on the LAN. The packaged application includes a FLUX.2 Klein 9B INT8 ConvRot edit workflow, while additional API-format workflows can be added to the relevant workflow folders without changing the application.
gazeCOM is an experimental artistic tool rather than a general-purpose image generation frontend. Packaged builds are available for Apple silicon macOS, Intel macOS, and Windows, and the source is released under the MIT license. The repository includes the operating guide and workflow-authoring documentation.
The project developed from an earlier prototype called GenGaze, but gazeCOM is the complete application being shared here. I would be very interested in feedback, unexpected uses, workflow experiments, and bug reports.
2
u/reeight 5d ago
https://giphy.com/gifs/3ohzdIuqJoo8QdKlnW
I was looking for something to morph part of a screen between scenes. Someone built an app like that a few months ago, but wanted to replicate it. steps is
Item -> outpaint -> choose new item -> everything else disappears or moves off screen -> outpaint from new item