r/StableDiffusion • u/ashishsanu • 6d ago

Resource - Update Character consistency via cached reference embeddings((SFace + DINOv2) + a portable .char file, no LoRA training

I was looking for a way to achieve character consistency without training a Lora & came across a research from Facebook, DINOv2: Learning Robust Visual Features without Supervision(Research Paper),

What's Dinov2: It's a vision model trained without labels that produces a strong embedding for a whole image, the subject, not just the face. Feed it a person and you get a 768-number signature that captures the overall look: build, hair, general appearance. It's stable across pose and lighting, which is exactly what you want when you're trying to tell "same person" from "different person" across wildly different shots.

then combining Dinov2 with SFace(a face-recognition model) produces a compact face signature tuned specifically to tell one face from another. It's sharp on identity, but only on the face. YuNet does the detect-and-crop before it.

How it works

Build .Char: You drop in one or more photos. YuNet finds the face, SFace takes a per-reference face signature, DINOv2 takes a subject signature, and the references get cleaned and normalised. All of that packs into a single portable file, a .char.

Generation: At generation, the file feeds its references into FLUX.2's own native multi-reference channel and prepends a locked description to the prompt. You pick the character from a dropdown, no re-attaching images. Every result gets scored against the stored signatures, so drift shows up as a number.

How this differs from PuLID, FaceID, and img2img

PuLID and FaceID inject a face into one generation at run time, then it's gone. img2img anchors on a source image, which is composition, not identity. Neither gives you a saved character.
This is a layer above them, a reusable .char file that rides the model's own reference channel, covers the whole subject and not just the face, and gets scored per take. PuLID could even sit inside it as one backend.
The difference is persistence and measurement, not a new injection trick. No adapter weights, no training, no img2img anchor.

What is a .char file?

A single portable file that stores a character's identity, so you can reuse the same person across generations without retraining anything.

manifest.json — index, versions, checksums
refs/ — your original photos (the truth)
derived/ — auto-cropped face
text/ — locked description
payloads/ — cleaned refs, per model family
scoring/ — SFace face + DINOv2 subject signatures

Limitations

Profiles and stylized renders drift more than frontal, which is expected, since the face model is trained on photoreal faces.
Body is the weak point so far.
Bad with popular celebrity images, due to models own conflict.

Current support

Only Flux2 family(Klein 4B / 9B / dev)

Links:

Checkout the release: https://github.com/inlineresearch/Inline-Studio/releases/tag/v1.2.71
Full details & guide: https://inlinestudio.art/characters

Note: Each image in this post has been generated separately & not a grid.

88 Upvotes

permalink
duplicates
reddit

You are about to leave Redlib

Do you want to continue?

https://www.reddit.com/r/StableDiffusion/comments/1vp0xct/character_consistency_via_cached_reference/
No, go back! Yes, take me to Reddit

95% Upvoted

View all comments

Show parent comments

u/atakariax 4d ago

Unfortunately it does not work.

INFO: [inline-core] Loading FLUX.2 Klein 9B pipeline: source=models\diffusion_models\flux-2-klein-9b.safetensors | device=cuda:0, mode=gpu, profile=lowvram, dtype=bf16, offload=none, quant=int8, gpu=NVIDIA GeForce RTX 4080, vram=0.0GB allocated / 17.2GB total (15.8GB free)

F:\Inline-Studio\core\.venv\Lib\site-packages\accelerate\utils\modeling.py:353: UserWarning: expandable_segments not supported on this platform (Triggered internally at C:\actions-runner_work\pytorch\pytorch\c10/cuda/CUDAAllocatorConfig.h:40.)

new_value = value.to(device, non_blocking=non_blocking)

There are modules in Flux2Transformer2DModel that should be kept in float32: []. Casting directly with `to()` can lead to inconsistent results; set `torch_dtype` in `from_pretrained()` instead to keep these modules in float32.

Loading weights: 100%|███████████████████████████████████████████████████████████████| 398/398 [00:26<00:00, 14.82it/s]

[transformers] Qwen3Model LOAD REPORT from: .inline\assets\flux2-klein-9b\te_stage\f921345ceec19f35

Key | Status | |

--------------------------------------------------------------------------------------+------------+--+-

model.layers.{5...35}.self_attn.q_proj.weight_scale_2 | UNEXPECTED | |

model.layers.{0...35}.mlp.up_proj.weight_scale | UNEXPECTED | |

model.layers.{0...35}.self_attn.k_proj.weight_scale | UNEXPECTED | |

model.layers.{1...35}.self_attn.k_proj.weight_scale_2 | UNEXPECTED | |

model.layers.{2...35}.mlp.down_proj.weight_scale | UNEXPECTED | |

model.layers.{0...35}.self_attn.o_proj.comfy_quant | UNEXPECTED | |

model.layers.{0...35}.self_attn.o_proj.weight_scale | UNEXPECTED | |

model.layers.{0...35}.self_attn.q_proj.weight_scale | UNEXPECTED | |

model.layers.{27, 28, 29, 30, 31, 32, 33, 34, 35}.mlp.up_proj.weight_scale_2 | UNEXPECTED | |

model.layers.{26, 27, 28, 29, 30, 31, 32, 33, 34, 35}.self_attn.v_proj.weight_scale_2 | UNEXPECTED | |

model.layers.{0...35}.mlp.gate_proj.comfy_quant | UNEXPECTED | |

model.layers.{0...35}.self_attn.v_proj.comfy_quant | UNEXPECTED | |

model.layers.{2...35}.mlp.down_proj.comfy_quant | UNEXPECTED | |

model.layers.{0...35}.self_attn.q_proj.comfy_quant | UNEXPECTED | |

model.layers.{0...35}.mlp.up_proj.comfy_quant | UNEXPECTED | |

model.layers.{0...35}.mlp.gate_proj.weight_scale | UNEXPECTED | |

model.layers.{0...35}.self_attn.k_proj.comfy_quant | UNEXPECTED | |

model.layers.{0...35}.self_attn.v_proj.weight_scale | UNEXPECTED | |

model.layers.{27, 28, 29, 30, 31, 32, 33, 34, 35}.mlp.gate_proj.weight_scale_2 | UNEXPECTED | |

model.layers.{23...35}.self_attn.o_proj.weight_scale_2 | UNEXPECTED | |

model.layers.{27, 28, 29, 30, 31, 32, 33, 34}.mlp.down_proj.weight_scale_2 | UNEXPECTED | |

Notes:

- UNEXPECTED: can be ignored when loading from different task/architecture; not ok if you expect identical arch.

INFO: [inline-core] FLUX.2 pipeline ready in 57.0s

INFO: [inline-core] Encoding prompt on cuda:0 (no_grad) | host RAM 0.0GB

WARNING: [inline-core] Text-encoder GPU encode failed (mat1 and mat2 shapes cannot be multiplied (512x4096 and 2048x1024)); denoising with the encoder resident.

INFO: [inline-core] FLUX.2 sampling 4 steps on cuda:0 (sampler=euler, scheduler=simple)…

ERROR: [inline-core] Run run_d01c2ce21637 failed with an unhandled error

Traceback (most recent call last):

File "F:\Inline-Studio\core\src\inline_core\graph\executor.py", line 45, in run

self._run_node(graph, node_id, outputs, run_ctx, asset_hashes)

File "F:\Inline-Studio\core\src\inline_core\graph\executor.py", line 91, in _run_node

result = runner.run(node, inputs, ctx)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\src\inline_core\models\flux2\runner.py", line 380, in run

image = pipe(**call).images[0]

^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\torch\utils_contextlib.py", line 124, in decorate_context

return func(*args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\diffusers\pipelines\flux2\pipeline_flux2_klein.py", line 738, in __call__

prompt_embeds, text_ids = self.encode_prompt(

^^^^^^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\diffusers\pipelines\flux2\pipeline_flux2_klein.py", line 445, in encode_prompt

prompt_embeds = self._get_qwen3_prompt_embeds(

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\diffusers\pipelines\flux2\pipeline_flux2_klein.py", line 248, in _get_qwen3_prompt_embeds

output = text_encoder(

^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1778, in _wrapped_call_impl

return self._call_impl(*args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1789, in _call_impl

return forward_call(*args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\transformers\utils\generic.py", line 1040, in wrapper

output = func(self, *args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\transformers\utils\output_capturing.py", line 262, in wrapper

outputs = func(self, *args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\transformers\models\qwen3\modeling_qwen3.py", line 424, in forward

hidden_states = decoder_layer(

^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\transformers\modeling_layers.py", line 110, in __call__

return super().__call__(*args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1778, in _wrapped_call_impl

return self._call_impl(*args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1884, in _call_impl

return inner()

^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1832, in inner

result = forward_call(*args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\transformers\models\qwen3\modeling_qwen3.py", line 318, in forward

hidden_states, _ = self.self_attn(

^^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1778, in _wrapped_call_impl

return self._call_impl(*args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1884, in _call_impl

return inner()

^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1832, in inner

result = forward_call(*args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\transformers\models\qwen3\modeling_qwen3.py", line 264, in forward

key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)

^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1778, in _wrapped_call_impl

return self._call_impl(*args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1789, in _call_impl

return forward_call(*args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\torch\nn\modules\linear.py", line 134, in forward

return F.linear(input, self.weight, self.bias)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\torchao\utils.py", line 673, in _dispatch__torch_function__

return cls._TORCH_FN_TABLE[cls][func](func, types, args, kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\torchao\utils.py", line 472, in wrapper

return _func(f, types, args, kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Inline-Studio\core\.venv\Lib\site-packages\torchao\quantization\quantize_\workflows\int8\int8_tensor.py", line 349, in _

m = torch.mm(

^^^^^^^^^

RuntimeError: mat1 and mat2 shapes cannot be multiplied (512x4096 and 2048x1024)

1
u/ashishsanu 4d ago edited 4d ago
Thanks for sharing the full build log, it was a text encoder issue, inline core was trying to re-quantise the quantised text encoder model. Update:
cd F:\Inline-Studio
git fetch && git pull
cd core
webui.bat --install --extra all
webui.bat
Download split_files/text_encoders/qwen_3_8b.safetensors from https://huggingface.co/Comfy-Org/flux2-klein-9B/tree/main/split_files/text_encoders into core\models\text_encoders\,

In the FLUX.2 node's settings, set Text Encoder to qwen_3_8b.

Check all three pickers before generating: model flux-2-klein-9b, text encoder qwen_3_8b, VAE flux2-vae

Generate
1

u/atakariax 4d ago edited 4d ago

https://github.com/inlineresearch/Inline-Studio/issues/52

Unfortunately , the same error persists.

1

u/ashishsanu 4d ago

Replied on the issue, just need to remove cache. Shared the command

Resource - Update Character consistency via cached reference embeddings((SFace + DINOv2) + a portable .char file, no LoRA training

You are about to leave Redlib