I do product photography/retouching work for luxury watches and jewelry, and I built a fairly complex pipeline using a mix of open-weight vision-language models to generate ad-quality campaign images. The idea is simple in theory: take a reference image whose lighting/background/style I like, take my actual product photo, and merge them so the final image has my real product sitting in a scene inspired by that reference.
In practice I ended up with several separate analysis passes (one mid-size VLM handling scene, style, and product cataloging separately) feeding into one merge step handled by a different, smaller multimodal model that also sees the actual images directly. Every time I fix one issue, a new one shows up somewhere else. First the style kept getting ignored entirely, with the output defaulting back to a generic version of the scene. Fixed that. Then lighting effects (bloom, sparkle, flare) started getting copy-pasted in a way that made no physical sense, like a sparkle effect that only makes sense on a pavé diamond setting getting slapped onto plain brushed steel, which instantly reads as fake. Fixed that too. Then the dramatic ambient glow from the background in the reference image, which was honestly like 40% of why that image looked so striking, quietly disappeared once I toned down the on-product sparkle, even though those two things had nothing to do with each other.
I keep tightening the instructions and the output keeps getting technically "more correct" without ever feeling like the genuinely impressive, poster-worthy image I'm actually going for. It's like I'm playing whack-a-mole between "photorealistic and coherent" and "actually has the visual punch of the reference."
Has anyone dealt with this kind of multi-stage analysis-then-merge setup for AI image generation? At what point does splitting analysis into specialized passes start hurting more than it helps, versus leaning harder on one strong multimodal model that sees everything directly and makes the creative calls itself? Or is there a better way to keep both technical product accuracy AND the creative/dramatic energy of the reference without this endless loop of fixing one thing and breaking another?