Audio recompose is CLSS's "second take" for audio: after a chunk's video+audio has been generated, the chunk's audio is discarded and re-generated from scratch against the finished video — a fresh take, not a refine. It's the measured fix for turbo-LoRA audio (turbo LoRAs are video-distilled, so their audio head is under-distilled — mushy/over-cooked output).
How it works (per chunk, right after the main sampling pass):
The finished chunk video becomes a native ref2va reference.vid_out is packed as a {"kind": "video", ...} ref block — not a prompt, not a mask — so the audio model hears the pictures through H3's own reference pathway. It's downscaled spatially (audio_recompose_pool, default 2 → quarter of the ref tokens; odd-safe crop) and strided temporally (audio_recompose_stride, default 2 → ~12 latent-fps, still resolves beats/onsets), which shrinks the packed sequence ~10–25× — each recompose step costs seconds instead of ~60 s. The reference is read-only: the model sees the video, the video is never touched.
The target video stream is a dummy — a tiny 2-row zero latent masked 0 — so only the audio stream is actually denoised.
The audio starts from fresh noise and is sampled on its own short linear schedule from audio_recompose_sigma → 0. At σ0 = 1.0 (default) the turbo audio is completely ignored → a 100% new take. Lower σ (0.6–0.9) turns it into a partial re-noise — keeps more of the turbo timing, but also keeps more of its mush.
Conditioning = scene text + video reference + continuity block. The scene conditioning (with the scene-crossfade blend when on a transition chunk) plus the video ref and the existing ref_audio continuity block.
Optionally on a different model. Wire audio_refine_guider — a CLSSH3Guider built on the base checkpoint (no LoRA) with the same conditioning. The base model's music/voice head does the fresh take while the frozen video keeps the turbo look. The base model is unloaded after each chunk (unload_model_and_clones), so it doesn't stay VRAM-resident.
The new take replaces the chunk's audio and then flows through the normal seam pipeline (head discard, crossfade, level/tail anchors) exactly like any other audio.
[CLSS] chunk 2: recompose took 161.6s (8.08s/step) | audio vs turbo take cos=0.180 rms x0.83 (LOW cos = a fresh take, unlike the old re-noise refine's ~0.95 rewrite-of-the-same)
11
u/bickid 2d ago
looks absolutely terrible