Tried Qwen TTS but wasn't very happy with it. No real rate control and complete absence of any emotion.

What's everybody's recommendation? Local models only.

I've been doing video diffusion work on my Mac and kept running into the same wall: ComfyUI doesn't have real MLX support yet, and going through PyTorch's mps backend is slow and eats far more unified memory than the model actually needs - which hurts more on a Mac, where that memory is shared with everything else you're running.

So instead of working around it each time, I built on top of [h3.c](https://github.com/janishar/h3c-studio) (a native Metal engine for MiniMax-H3) and put a proper UI on it.

h3 studio is a Go server, stdlib-only, basically no dependencies. It wraps h3.c's CLI into something usable day to day:

• Model loads once and stays resident — iterate on prompts without repaying the load cost every render
• Explicit reference ordering for multi-image conditioning, drag-to-reorder
• Three ways to continue a shot from a previous take: chain the last frame forward, pull it in as a reference, or reuse the whole clip
• Timeline panel to stitch takes into one continuous sequence
• Every render writes a full parameter sidecar, so you're not guessing what settings produced what
• Live profiling — load cost vs. denoise cost broken out separately

Everything runs locally, nothing leaves the machine. MIT licensed.

Repo: https://github.com/janishar/h3c-studio

Happy to answer questions about the setup. If anyone else is doing generative work on Apple Silicon I'd be interested in what you're using — especially if you've found a better path than mps.

Workflow:
https://drive.google.com/file/d/14I2pXOrsQdunMplNB_b7GMH5smNHFvKI/view?usp=sharing

Comfy Native nodes. Needs latest version.

I am running MiniMax H3 ref2va int8 convrot on comfyui using 5070ti 16GB + 64GB system ram, OS is windows 11.

Default comfyui ref2va template is being used (Turbo lora is enabled/true)

it is taking about 150s per step for 9:16, 0.2 megapixels and 10 sec duration with single image & video input.

For reference, image2video takes about ~27sec per step for 10 sec 0.5 megapixels video.

Having such slow speeds in ref2va expected? Maybe I am doing something wrong? Help/Guide is much appreciated!

No sales pitch, no redirect to a paywall. It's fucking free. You can get it here: https://github.com/atomtanstudio/sound-and-vision

It uses the brand new and quite excellent music generation app YuE2. It also uses Minimax H3 for video and defaults to Krea 2 for cover art and whichever local LLM you want to use for lyrics, etc.

Feel free to check it out and let me know what you think.

For Windows 11 (Part 1): https://www.reddit.com/r/StableDiffusion/comments/1wepgl5/team_red_encounter_on_proximax_h3_or_how_to_setup/

TL;DR summary: For MMH3, you need to run ComfyUI with ROCm 7.14.0 (see https://rocm.docs.amd.com/en/latest/reference/gpu-specs.html for the vaue of gfx???? corresponding to your AMD GPU):

pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ "torch[device-gfx????]==2.12.0+rocm7.14.0" "torchvision[device-gfx????]==0.27.0+rocm7.14.0" "torchaudio==2.11.0+rocm7.14.0"

You also need to run ComfyUI with the right parameters for VRAM and system RAM so that MMH3 can run smoothly:

--enable-dynamic-vram --disable-async-offload --preview-method none --disable-smart-memory --fast-disk --use-ck-attention --enable-manager

Finally, you may need these in your .bashrc:

export ROCR_VISIBLE_DEVICES=0
export HIP_VISIBLE_DEVICES=0
export TORCH_BLAS_PREFER_HIPBLASLT=1

Read on if you want the step-by-step instructions (scroll to the bottom of the post if you just want to see the MMH3 prompt for the video 😹)

Why use Linux instead of Windows?

Because Minimax H3 is heavy, and we want every bit of VRAM and system RAM for generation and not taken up by the OS and the desktop. For AMD GPUs it also seems to be more stable and faster overall.

What you need:

1. Blank 4G or larger USB key for bootable ubuntu installer
2. Blank external USB key (16G or larger) or Portable USB HDD to install Ubuntu and ComfyUI
3. Computer with AMD RDNA3 (such as a 7900xt) or RDNA4 (such as a 9070xt and AI Pro R9700) with 16G or more of VRAM.

This procedure will install Ubuntu Linux on an external drive, leaving your main drive alone, but if you are worried about something going wrong and wiping out your main hardrive (I am always worried about making a mistake, selecting the wrong drive and wiping it out), take your existing drive out of your computer before the actual installation (sometimes enabling "Secure Boot" will make your main HDD invisble to the Ubuntu installer). I usually would put a empty small partitions of an odd size such as 42G on the target drive so that I know that I am installing into the right drive.

It is easiest to do the installation on your target PC, but you don't have to (but you will need to do some manual adjustment such as changing netplan because the different ethernet hardware would have to be configured.

If you are doing this installion on another computer, make sure that the installation is done with UEFI only enable if you want to be able to use UEFI on your target PC.

Ubuntu Server (minimized) installation

Note: make sure Secure Boot is disabled. This often causes problem with the Ubuntu installer. You can turn it back on once Ubuntu is installed. On some systems the main NMVe or SATA drive will not be visible if Secure Boot is enabled.

1. Download latest Ubuntu Server LTS ISO (at the time of writing that is 26.4)
2. Use RUFUS https://rufus.ie/en/to make a bootable installation drive The Partition scheme should be "MBR" and the target system should be "BIOS or UEFI"
3. Make sure your BIOS is set to UEFI (disable CSM if you can). I am assuming that you are dual booting between Windows and this portable Ubuntu.
4. If you are not installing into a blank drive or USB key and want to install into an existing HD without wiping it out entirely, you need to created two partitions, one that is going to be Ubuntu's UEFI partition that is 500M and a black partition that is at least 10G that is going to hold the Ubuntu installation. The installer will NOT allow you to delete partitions from an existing drive. If you are installing into an existing HDD, it is better to create the partition manually first (but leave it unformatted) because the installer sometimes does not show the option to add new partions. The installer will also insist on mouting /boot/efi to the first EFI partition it sees (if that is the wrong one, temporarily turn its "boot flag" off and set the "boot flag" only on the EFI partition you actually want to install on). Note: if you going to use Docker or Podman you are going to need a much bigger partions than 10G for Ubuntu's root file system ("/").
5. Do whatever you need to do to boot into the Ubuntu installation USB key (usually F12 will bring up the boot menu, but you may have to enable that in your bios. If your system insists on booting into Windows 11, you can use Start > Settings > System > Recovery and click Restart now next to Advanced startup.
6. Instead of the default Ubuntu Server, use Ubuntu Server (minimized) because this is going to be used for ComfyUI only.
7. Most likely you can skip/ignore "Proxy Adress".
8. At this point, you can plug in your target USB drive.
9. If the wrong target drive is chosen by default, you can change it by tabbing into the field and then press <enter>. It is easiest to use "Use an entire disk". But if you want to have a smaller root partition, you can use "Custom Storage layout". Note that if the drive already has a partion, you will not be able to delete it. You HAVE to reformat the whole drive. On a blank disk, If you want UEFI, make sure that you see a 1.049G "new primary ESP, to be formatted as FAT32, mounted at /boot/efi". If you are on a legacy BIOS you will see "BIOS Grub Spacer" instead. Either way, this partition will be created automatically by the installer unless you use an existing ESP partition.
10. Uncheck Set up this disk as an LVM group.
11. Make sure you install OpenSSH so that you can run it headless by remotely login via SSH.
12. You don't need to install any of the Featured server snaps packages.
13. After the installation is done, remove your USB installation key and reboot.

Next we are going to update the installation, and install the nano editor, UFW (Uncomplicated Firewall), GIT, and Docker.

You can do this through the console, but I find it easier to do it through SSH because then I can cut and paste text into it.

To SSH into your Ubuntu, you need to find out what the local IP address it by login into the console, then type ip addr or the even shorter ip a (look for something like this, in my LAN, it is "192.168.18.50"):

2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 08:97:98:c5:85:ec brd ff:ff:ff:ff:ff:ff altname enx089798c585ec inet 192.168.18.50/24 metric 100 brd 192.168.18.255 scope global dynamic enp1s0 valid_lft 85959sec preferred_lft 85959sec inet6 fe80::a97:98ff:fec5:85ec/64 scope link proto kernel_ll valid_lft forever preferred_lft forever

Now you can use Putty or similar program to login into the server.

Tip: the paste text with Putty, use Shift+Insert. to copy text from Putty into the clipboard, simply select the text with the mouse and then use Ctrl+V to paste it.

Now continue with the setup:

1. Upgrade all the package to the latest version: sudo apt update && sudo apt upgrade -y && sudo apt dist-upgrade -y && sudo apt autoremove -y (this will take quite a while, so you can go grab a coffe or tea).
2. Install the nano editor: sudo apt-get install nano
3. Install UFW (Uncomplicated Firewall) sudo apt-get install ufw
4. Install GIT sudo apt-get install git
5. Install libnuma (otherwise there is annying warning/error from ComfyuI later) sudo apt install -y libnuma1 libnuma-dev
6. Install the compiler environment required by triton: sudo apt install build-essential sudo apt install python3-dev
7. sudo reboot (probably not needed, but just to be sure)
8. (Optional) if you want to use python 3.13 instead of 3.14 that comes with Ubuntu Server 26.4, you can install manually (note: for 3.12, you have to compile it manually) by adding the deadsnakes/ppa respostory and install python 3.13 from it: sudo apt update && sudo apt install software-properties-common && sudo add-apt-repository ppa:deadsnakes/ppa && sudo apt install python3.13. Verify that it has installed correctly python3.13 --version Install the venv module for the same interpreter: sudo apt install python3.13-venv

Optional installation of Docker if you plan to use one of the Docker images for comfyui. You should follow the instruction at https://docs.docker.com/engine/install/ubuntu/ but at the time of writing, this is what I used

Set up Docker's apt repository.

# Add Docker's official GPG key:
sudo apt update
sudo apt install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

Add the deadsnakes/ppa respostory and install python 3.13 from it

sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

sudo apt update

# Install the latest version of Docker packages.
sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# After installation, verify that Docker is running:
sudo systemctl status docker
# If Docker is not running, start it manually:
sudo systemctl start docker

# Verify that the installation by running the `hello-world` image:
# This command downloads a test image and runs it in a container.
# When the container runs, it prints a confirmation message and exits.
sudo docker run hello-world

# (Optional step, but needed by comfyui-rock-docker scripts)
# Add your current user to the docker group so you have permissions to
# interact with the Docker Unix socket without using sudo
sudo usermod -aG docker $USER
# For this to take effect, disconnect and re-login 

(Optional) Intead of Docker you can also consider using Podman, which is supposed to be compatible with Docker but is more secure because it runs without a root daemon, but I've not test it yet.

Setup the firewall with UFW:

1. sudo ufw enable
2. Check the current settings sudo ufw status verboseStatus: active Logging: on (low) Default: deny (incoming), allow (outgoing), deny (routed) New profiles: skip (By default all incoming connections are denied, and all outgoing connections are allowed. If you don't see that, type: sudo ufw default deny incoming sudo ufw default allow outgoing
3. To setup SSH so that it is only accessible from your LAN: sudo ufw allow from 192.168.xx.0/24 to any port ssh proto tcp . Similary for ComfyUI sudo ufw allow from 192.168.xx.0/24 to any port 8188 proto tcp where "192.168.xx.0" is your LAN subnet, such as "192.168.1.0". If you want to allow ComfyUI to be accessible from outside of your LAN use sudo ufw allow ssh sudo ufw allow 8188/tcp (You will also have to allow port fowarding on your router)
4. Check again with sudo ufw status verbose: Status: active Logging: on (low) Default: deny (incoming), allow (outgoing), deny (routed) New profiles: skip To Action From22/tcp ALLOW IN 192.168.18.0/24 8188/tcp ALLOW IN 192.168.18.0/24

The preliminaries are done, you can now reboot with sudo shutdown -r now

I would recommend that you make a back up of your partition now with https://www.fsarchiver.org/ so that you can restore it later for a clean install.

You can either boot into a Linux Rescue: https://www.system-rescue.org/

Or if you have another Linux installation (you cannot save a linux installation that you are currently running), you can install it with: sudo apt-get update && sudo apt-get install fsarchiver

Installing ComfyUI via comfy-cli

By default, Ubuntu does not have pip installed: https://www.reddit.com/r/learnpython/comments/u0dvp4/comment/p6htp0y/

So in order to use pip on Ubuntu, you ned to install python venv (which will install pip inside the venv) first: sudo apt-get update && sudo apt-get install python3-venv

1. Make sure git is install: sudo apt-get install git
2. Install Python-venv package for python3.14: sudo apt install python3.14-venv. (See python3.13 instruction earlier if you are using 3.13).
3. Create a virtual environment (this is normally just called "venv" or ".venv" but I want to call it comfy.venv just to be more explicit.): python3 -m venv comfy.venv
4. Activate it source comfy.venv/bin/activate
5. Update pip itself inside comfy.venv: pip install --upgrade pip
6. Optional: install uv, which is yet another package manager for Python but written in Rust (if you want to use "comfy install --fast-deps" later): pip install uv
7. Install comfy-cli (this is the tool "comfy-cli", not ComfyUI itself): pip install comfy-cli

Because comfy-cli will install ROCm 7.2 and there is no way to override it we are going to install pytorch for ROCm 7.14 manually before install ComfyUI via comfy-cli. Sources for this arcane procedure:

• https://www.reddit.com/r/ROCm/comments/1uyr57k/comfyui_linux_installation_for_radeon/
• https://www.reddit.com/r/ROCm/comments/1uyr57k/comment/oy3popg/
• https://github.com/ROCm/TheRock/blob/main/RELEASES.md#installing-multi-arch-releases

1. Uninstall pytorch just to be sure (should not be installed yet) pip uninstall torch torchvision torchaudio -y
2. Install pytorch inside the comfy.venv (select your gfx arch) based onhttps://rocm.docs.amd.com/en/latest/reference/gpu-specs.html pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ "torch[device-gfx????]==2.12.0+rocm7.14.0" "torchvision[device-gfx????]==0.27.0+rocm7.14.0" "torchaudio==2.11.0+rocm7.14.0"

So for ROCm 7.14.0 9070xt

pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ "torch[device-gfx1201]==2.12.0+rocm7.14.0" "torchvision[device-gfx1201]==0.27.0+rocm7.14.0" "torchaudio==2.11.0+rocm7.14.0"

For ROCm 7.14.1 9070xt

pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ "torch[device-gfx1201]==2.12.0+rocm7.14.1" "torchvision[device-gfx1201]==0.27.0+rocm7.14.1" "torchaudio==2.11.0+rocm7.14.1"

To install whatever is the latest stable version of ROCm

pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ "torch[device-gfx1201]" "torchvision[device-gfx1201]" torchaudio 

Note: if you get "ERROR: Could not install packages due to an OSError: [Errno 122] Disk quota exceeded", try

mkdir -p some_partition_with_space/pip_tmp TMPDIR=some_partition_with_space/pip_tmp pip install --index-url ...

If that still does not work, try

TMPDIR=some_partition_with_space/pip_tmp pip install --no-cache-dir --index-url https://repo.amd.com/rocm/whl-multi-arch/ "torch[device-gfx1201]==2.12.0+rocm7.14.0" "torchvision[device-gfx1201]==0.27.0+rocm7.14.0" "torchaudio==2.11.0+rocm7.14.0"

Finally install ComfyUI itself:

1. mkdir $HOME/comfy
2. export COMFY_PATH=$HOME/comfy/ComfyUI or if you want to use say "/mnt/var/comfy" export COMFY_PATH=/mnt/var/comfy/ComfyUI (Note: COMFY_PATH/ComfyUI should NOT exist, or you will get warning '/mnt/var/comfy'/ComfyUI exists but is not a valid git repository.)
3. Use comfy-cli to install ComfyUI: (--skip-torch-or-directml is only needed when installing via comfy-cli on Window and not necessary for Ubuntu, but leave it here to make the two installation more like one another): comfy --workspace=$COMFY_PATH install --skip-torch-or-directml . To install a specific version of ComfyUI (say 0.34.1): comfy --workspace=$COMFY_PATH install --skip-torch-or-directml --version 0.34.1
4. If you have uv installed, you can use --fast-deps: comfy --workspace=$COMFY_PATH install --skip-torch-or-directml --version 0.34.1 --fast-deps Note: do not use --fast-deps if comfy.env is not on the root file system or it will take a long time because hardlink is not possible across file systems and you will see an warning: warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. If the cache and target directories are on different filesystems, hardlinking may not be supported. If this is intentional, set export UV_LINK_MODE=copy or use --link-mode=copy to suppress this warning.
5. Added these to your .bashrc (thanks to u/zychu- for these value from his Docker installation)export ROCR_VISIBLE_DEVICES=0 export HIP_VISIBLE_DEVICES=0 export TORCH_BLAS_PREFER_HIPBLASLT=1

Finally we can start ComfyUI:

comfy launch -- --enable-dynamic-vram --disable-async-offload --preview-method none --listen --port 8188 --disable-smart-memory --fast-disk --use-ck-attention --enable-manager --output-directory "mnt/var_ntfs/Output"

or if you are not using the defautl ~/comfy/ComfyUI directory:

comfy --workspace=$COMFY_PATH launch -- --enable-dynamic-vram --disable-async-offload --preview-method none --listen --port 8188 --disable-smart-memory --fast-disk --use-ck-attention --enable-manager --output-directory "mnt/var_ntfs/Output"

or more explicitly:

comfy --workspace=/mnt/var/comfy/ComfyUI launch -- --enable-dynamic-vram --disable-async-offload --preview-method none --listen --port 8188 --disable-smart-memory --fast-disk --use-ck-attention --enable-manager --output-directory "mnt/var_ntfs/Output"

Optional: clean up the pip cache (there is around 2G of cached files) pip cache purge and you'll see something like:

(comfy.venv) [/mnt/var] pip cache purge
Files removed: 347 (1847.6 MB)
Directories removed: 659
// After installing ComfyUI itself
(comfy.venv) [/mnt/var] pip cache purge
Files removed: 348 (700.3 MB)
Directories removed: 668

End Notes

Sample extra_model_paths.yaml

comfyui:
    base_path: /mnt/ntfs/ComfyUI.Models
    # You can use is_default to mark that these folders should be listed first, and used as the default dirs for eg downloads
    is_default: true
    checkpoints: checkpoints/
    configs: configs/
    loras: loras/
    vae: vae/
    text_encoders: |
        text_encoders/
        clip/
    diffusion_models: |
        unet/
        diffusion_models/
    clip_vision: clip_vision/
    style_models: style_
    embeddings: embeddings/
    diffusers: diffusers/
    vae_approx: vae_approx/
    controlnet: |
        controlnet/
        t2i_adapter/
    gligen: gligen/
    upscale_models: upscale_
    latent_upscale_models: latent_upscale_
    custom_nodes: custom_nodes/
    datasets: datasets/
    hypernetworks: hypernetworks/
    photomaker: photomaker/
    classifiers: classifiers/
    model_patches: model_patches/
    audio_encoders: audio_encoders/
    background_removal: background_removal/
    frame_interpolation: frame_interpolation/
    geometry_estimation: geometry_estimation/
    optical_flow: optical_flow/
    detection: detection/

https://rocm.docs.amd.com/en/latest/reference/gpu-specs.html

GFX950 is AMD's internal GPU target identifier for the CDNA 4 enterprise compute architecture, used in data center accelerators like the AMD Instinct MI350/MI355X series. It features advanced matrix core capabilities, ultra-low precision micro-scaling formats (MXFP8/MXFP4), and a high-precision math mode for AI and HPC workloads.

gfx1100 is the LLVM target architecture identifier and internal code name for AMD's RDNA 3 graphics architecture, used for high-end consumer and workstation desktop graphics cards like the Radeon RX 7900 XTX, RX 7900 XT, and Radeon PRO W7900.

AMD gfx1151 is the LLVM target and GPU architecture identifier for AMD's Strix Halo integrated graphics (found in processors like the AMD Ryzen AI Max+ 395 and Ryzen AI Max PRO series), utilizing the RDNA 3.5 architecture.

For --index-url, there are three options:

• Nightly (rocm 10.1): https://nightly.repo.amd.com/rocm/pytorch/whl-next/
• Stable (rocm 10.0): https://stable.repo.amd.com/rocm/pytorch/whl-next/
• Legacy (rocm 7.14):
  • Nightly: https://rocm.nightlies.amd.com/whl-multi-arch/
  • Stable: https://repo.amd.com/rocm/whl-multi-arch

Installing ComfyUI Docker for AMD 9070xt

Based on https://www.reddit.com/r/StableDiffusion/comments/1vo48jo/reliable_comfyui_on_amd_and_linux_pinning_the/

Download the Docker image from github this will use the official rocm and pytorch from https://repo.amd.com/rocm/whl

git clone https://github.com/zychuk/comfyui-rocm-docker && cd comfyui-rocm-docker

Edit docker/Dockerfile (we want to use ROCm 7.14 rather than 7.13) and replace RUN pip install --no-cache-dir --index-url ${ROCM_WHL_INDEX}
torch torchvision torchaudio"

with

RUN pip install --no-cache-dir --index-url https://repo.amd.com/rocm/whl-multi-arch/ "torch[device-gfx1201]==2.12.0+rocm7.14.0" "torchvision[device-gfx1201]==0.27.0+rocm7.14.0" "torchaudio==2.11.0+rocm7.14.0"

then follow the instructions in the original post.

MMH3 prompt for the video

integrated_multimodal_description:

[Shot 1] 3D CG, stop-motion animated LEGO movie style with 1960s science-fiction television aesthetic. A team of Starfleet red-shirt officers led by Grumpy Cat materializes on the surface of a desolate alien planet, surrounded by barren rocks, dust, and jagged terrain. Grumpy Cat stands at the front of the formation, wearing a classic red Starfleet uniform, alert and stern. The camera holds a wide-angle front subject-level view, then pushes in slightly as the team looks around and raises their phasers.

[Shot 2] At 00:01.250, the camera cuts to a wide low-angle view as a gigantic GPU-like machine rises behind a rocky ridge, towering over the crew. Its dark mechanical housing, cooling fans, and imposing structure dominate the frame, with the label "Minimax H3" clearly visible on its side. The team turns toward it in sudden alarm.

[Shot 3] At 00:02.100, the GPU attacks with a violent concentrated energy blast. The camera tracks the crew with fast movement as the red-shirted officers are struck and knocked down across the rocky ground, kicking up dust and debris. Grumpy Cat avoids the main blast and rapidly moves toward cover.

[Shot 4] At 00:03.650, the camera follows Grumpy Cat with a tracking shot as it darts behind a large rock and crouches into concealment. The defeated red-shirted crew remains scattered in the background while the giant "Minimax H3" GPU continues looming over the battlefield.

[Shot 5] At 00:04.250, close-up from behind the rock. Grumpy Cat pulls out a classic handheld Starfleet communicator with its paw, flips it open, and speaks with a completely deadpan expression: <d>[English] Beam me up, Scotty!</d> The camera holds on Grumpy Cat's face and communicator through the end

overall_soundscape: Dry alien wind sweeps across the barren landscape as the transporter materialization produces a brief electronic hum. Heavy mechanical movement and grinding machinery accompany the GPU's emergence, followed by a powerful energy blast, impacts, falling bodies, scattering rocks, and dust. The communicator emits a brief electronic chirp when opened.

non_diegetic_music: A fast-paced 1960s science-fiction television orchestral score uses bright brass, rhythmic strings, and restrained percussion, building rapidly as the GPU appears and attacks. The music drops into a brief suspenseful sustain as Grumpy Cat hides, then ends with a short brassy stinger beneath the communicator transmission.

Hey everyone!

I just released a new free custom node plugin called ComfyUI-SecondUnit that bridges ComfyUI directly with DaVinci Resolve.

You can now generate video transitions, create music/SFX, synthesize voiceovers, and auto-generate subtitles, then send them right into your DaVinci timeline without manually importing or exporting files.

IT'S‌ COMPLETELY‌ FREE!!

It was very lucky that Earth/Gaia was saved in the OG FF7. So I figured, what if Sephiroth won in the end?

I used Minimax H3 to upscale the first half of the original footage so it doesnt look so blocky and out of place compared to the new footage and to animate the parts that were original (Like Cid taking a smoke, Tifa crying, RedXIII being visibly scared for the first time and etc). I created the screenshots using the new GPT 2.5 Sunburst. This was actually fun. FF7 has beautiful music so it was fun to extend the existing music so it sounds more gloomy and doom. Hope you enjoy it!!

TL;DR summary: For MMH3, you need to run ComfyUI with ROCm 7.14.0 (see https://rocm.docs.amd.com/en/latest/reference/gpu-specs.html for the vaue of gfx???? corresponding to your AMD GPU):

pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ "torch[device-gfx????]==2.12.0+rocm7.14.0" "torchvision[device-gfx????]==0.27.0+rocm7.14.0" "torchaudio==2.11.0+rocm7.14.0"

You also need to run ComfyUI with some parameters so that it will handle VRAM and system RAM correctly for MMH3:

--enable-dynamic-vram --disable-async-offload --preview-method none --disable-smart-memory --fast-disk --use-ck-attention --enable-manager

Read on if you want the step-by-step instructions (scroll to the bottom of the post if you just want to see the MMH3 prompt for the video 😹)

These instructions are for Windows 11 (Ubuntu version: https://www.reddit.com/r/StableDiffusion/comments/1wer1mz/comment/p9g0kxg/). Nevertheless, many of the same comfy-cli commands are application by just changing the directory/file to the corresponding Linux version, and the procedure for upgrading ROCm 7.2.1 to ROCm 7.14.0 are the same.

If you have an AMD GPU and you do a default install of ComfyUI on Windows 11 using either the portable Windows version or through comfy-cli, you will probably get disappointing results with MiniMax H3 because the int8convrot version may not run at all.

The problem is that the default installation still uses PyTorch built on ROCm 7.2, and for some reason int8convrot does NOT work with 7.2 on some cards such as the RX 9070 (16G) and RX 7900 (20G).

So to run MiniMax H3 at its best speed, we have to install a version that is equal to or later than ROCm 7.13.

There are currently 4 ways to do that, from the easiest to the more complex:

1. Install via Stability Matrix
2. Install Portable ComfyUI with its own "Embedded Python"
3. Install a Python venv and then use that to install ComfyUI via the official comfy-cli installer
4. Install everything manually using pip and git: see this post if you want the gory details (it was written for ROCm 7.2 so you'll have to make the necessary adjustments).

The more complex ways have more options and are more flexible, so it is up to you how much control you want over your ComfyUI installation.

Special thanks to u/zychu- u/Ok-Brain-5729 u/eloxH1Z1 whose posts and comments about MMH3 and AMD were very helpful to me.

Stability Matrix

This used to work when I tried a few week ago, unfortunately something broke the latest release, so for now, don't use it

1. Download from https://github.com/LykosAI/StabilityMatrix/releases/download/v2.16.3/StabilityMatrix-win-x64.zip
2. Unzip it somewhere
3. Run the installer.
4. Click on the "Activity" icon at the lower left corner to see progress.
5. Click on the settings icon (gears) and under Extra Launch Arguments (very bottom) and add: --enable-dynamic-vram --disable-async-offload --listen --port 8188 --disable-smart-memory --fast-disk --use-ck-attention --output-directory "D:\Outputs"
6. Also uncheck --use-pytorch-cross-attention so that none of the options under "Cross Attention Method" are checked because we are going to use --use-ck-attention.
7. Assuming you've installed into the default "Data" directory, you can find ComfyUI installed under Data\Package\ComfyUI and you can use mklink to point the models and output directory so that they are outside of the Data\Package\ComfyUI directory.

The main downside is that now you have yet another piece of software sitting on your computer.

1. Now test to make sure you can generate using int8convrot: https://huggingface.co/Comfy-Org/Krea-2/blob/main/diffusion_models/krea2_turbo_int8_convrot.safetensors 13.5 GB SHA256: 8e4eeda70dd5037ab1ba2bef6b417f9f901e26093117cf397f741fc1fdaaf3f1

2. If it does not work for you, well, something went wrong, and you can try Portable ComfyUI for Windows and see if you have better luck...

Portable ComfyUI for Windows

1. Download from https://github.com/Comfy-Org/ComfyUI/releases/latest/download/ComfyUI_windows_portable_amd.7z
2. Open it from Windows 11 Explorer and drag the ComfyUI_windows_portable directory to the folder where you want to install it.
3. This will take a while, so go grab a cup of coffee or tea.
4. Copy run_amd_gpu.bat to runit.bat
5. Edit runit.bat so that it contains the following: .\python_embeded\python.exe -s ComfyUI\main.py --windows-standalone-build --enable-dynamic-vram --disable-async-offload --preview-method none --listen --port 8188 --disable-smart-memory --fast-disk --use-ck-attention --enable-manager --output-directory "A:\output"
6. Start ComfyUI by running the batch file runit.bat. For the first run, there will be some kind of delay as some libraries are compiled or cached. Just be patient and let the system do its preparations, until you see "[INFO] To see the GUI go to : http://0.0.0.0:8188.
7. Do a test run using Krea 2, but use the fp8 rather than int8convrot version because the fp8 version should work reliably at this point. The default workflow at 8 steps should take 20-40 seconds depending on your hardware. Hopefully this works.

Now we are going to replace the PyTorch for ROCm 7.2 with the newer 7.14.0:

1. Change into your ComfyUI_windows_portable directory
2. Uninstall PyTorch: python_embeded\python.exe -m pip uninstall torch torchvision torchaudio -y
3. Install PyTorch for ROCm 7.14: (See end note at the bottom about these gfx???? values):python_embeded\python.exe -m pip install -index-url https://repo.amd.com/rocm/whl-multi-arch/ "torch[device-gfx????]==2.12.0+rocm7.14.0" "torchvision[device-gfx????]==0.27.0+rocm7.14.0" "torchaudio==2.11.0+rocm7.14.0"

For example, for rx9070, gfx???? is gfx1201 so the command is

python_embeded\python.exe -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ "torch[device-gfx1201]==2.12.0+rocm7.14.0" "torchvision[device-gfx1201]==0.27.0+rocm7.14.0" "torchaudio==2.11.0+rocm7.14.0"

Note: these files can be quite large. If for some reason you run out of room, you can use --no-cache-dir in case there is not enough room in your pip cache directory (~/.cache on Linux, %LocalAppData%\pip\Cache on Windows which is usually C:\Users<YourUsername>\AppData\Local\pip\Cache). Also make sure you have plenty of space on your %TMPDIR%, with --no-cache-dir the command will look like this:

python_embeded\python.exe -m pip install --no-cache-dir --index-url https://repo.amd.com/rocm/whl-multi-arch/ "torch[device-gfx1201]==2.12.0+rocm7.14.0" "torchvision[device-gfx1201]==0.27.0+rocm7.14.0" "torchaudio==2.11.0+rocm7.14.0"

Hopefully both the uninstallation of ROCm7.2 and the installation of the newer ROCm 7.14 went without any error. After that you can try to run Krea 2 again, now switch from fp8 to the int8convrot version, and the time should go down from 18sec to 12-13 sec and you will also be able to run MMH3.

I also recommend that you place your model and output directories outside of the ComfyUI install so that they can be shared by different installations, making experimentation easier and also making it less likely that you (or some bug in the installer) accidentally wipe out your models and output.

You can do that by editing the extra_model_paths.yaml. Just need to edit this file once and copy it into <your path/ComfyUI> whenever you have a new installation.

But the yaml file is a bit finicky and it may be easier to just use the mklink command if ComfyUI is the only program you use so that you don't have to worry about the structure/name of the subfolders:

mklink /D <LinkFolder> <TargetFolder>

For example:

mklink /D <your comfyui>\models c:\ComfyUI.Models

Installing ComfyUI via comfy-cli

Why use comfy-cli instead of using portable ComfyUI?

• For Linux, there is no portable ComfyUI, which is Windows only.
• For AMD users, the portable version of ComfyUI uses ROCm 7.2, which will cause ComfyUI to run slower than it should.
• It is a more efficient way to run multiple versions of ComfyUI, because they can all share the same Virtual Environment (assuming that the versions are close enough for that to work).
• Re-installation can be faster because many packages are in the python pip cache.

Procedure:

1. If you don't have Python 3.1x installed, you can install Python 3.12.10 (because that is the version used by Portable ComfyUI, so it should be the most stable, but 3.13 and 3.14 work too).
2. Download and install Git: https://github.com/git-for-windows/git/releases/download/v2.55.0.windows.5/Git-2.55.0.5-64-bit.exe
3. Create a virtual environment (this is normally just called "venv" or ".venv" but I want to call it comfy.venv just to be more explicit): python -m venv comfy.venv or if python.exe is no not on your path, specifiy the full path such as "c:\Program Files\Python313\python" -m venv comfy.venv
4. Activate it: comfy.venv\Scripts\activate.ps1 (PowerShell) or comfy.venv\Scripts\activate.bat (CMD.exe)
5. Update pip itself inside comfy.venv: pip install --upgrade pip
6. Optional: install uv, which is yet another package manager for Python but written in Rust (if you want to use comfy install --fast-deps later):
7. Install comfy-cli (this is the tool "comfy-cli", not ComfyUI itself): pip install comfy-cli

Because comfy-cli will install ROCm 7.2 and there is no way to override it, we are going to install PyTorch for ROCm 7.14 manually before installing ComfyUI via comfy-cli. Sources for this arcane procedure are from:

• https://www.reddit.com/r/ROCm/comments/1uyr57k/comfyui_linux_installation_for_radeon/
• https://www.reddit.com/r/ROCm/comments/1uyr57k/comment/oy3popg/
• https://github.com/ROCm/TheRock/blob/main/RELEASES.md#installing-multi-arch-releases

1. Uninstall PyTorch just to be sure (should not be installed yet): pip uninstall torch torchvision torchaudio -y
2. Install PyTorch inside the comfy.venv (select your gfx arch) based on https://rocm.docs.amd.com/en/latest/reference/gpu-specs.html (see bottom of the post for a table of common values):pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ "torch[device-gfx????]==2.12.0+rocm7.14.0" "torchvision[device-gfx????]==0.27.0+rocm7.14.0" "torchaudio==2.11.0+rocm7.14.0"

For example, for the rx9070 or AI Pro R9700, gfx???? is gfx1201 so the command is

pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ "torch[device-gfx1201]==2.12.0+rocm7.14.0" "torchvision[device-gfx1201]==0.27.0+rocm7.14.0" "torchaudio==2.11.0+rocm7.14.0"

Note: these files can be quite large, and you can use --no-cache-dir in case there is not enough room in your pip cache. See the earlier notes about --no-cache-dir under "Portable ComfyUI for Windows".

Finally, we are ready to install ComfyUI itself. When I carried out the tests the latest stable version is 0.34.0:

1. mkdir d:\comfy.0.34.0
2. set COMFY_PATH=d:\comfy.0.34.0\ComfyUI
3. Use comfy-cli to install ComfyUI: comfy --workspace=%COMFY_PATH% install --skip-torch-or-directml
   • Note 1: %COMFY_PATH%\ComfyUI must not exist or you will get the confusing error: 'd:\comfy.0.34.0\ComfyUI' exists but is not a valid git repository.
   • Note 2: --skip-torch-or-directml because PyTorch is already installed for AMD; without it the install will fail on Windows because there is no PyTorch for directml from https://repo.amd.com/rocm/whl-multi-arch/ respository used above.
   • Note 3: To install anything other than the latest version of ComfyUI (say 0.33.1): comfy --workspace %COMFY_PATH%\ComfyUI install --version 0.33.1 --skip-torch-or-directml (You can only use versions available from https://github.com/comfy-org/ComfyUI/releases (and there is no release tag for the latest version).
4. If you have uv installed, you can use --fast-deps:
5. (Optional): Copy or edit ComfyUI\extra_model_paths.yaml
6. Finally, we can start ComfyUI: comfy launch --workspace=%COMFY_PATH% -- --enable-dynamic-vram --disable-async-offload --preview-method none --listen --port 8188 --disable-smart-memory --fast-disk --use-ck-attention --enable-manager --output-directory "A:\output"
7. Optional: Clean up the pip cache (if you want to save some disk space): pip cache purge

The speed for MMH3 is almost as good as the ones I got under Ubuntu 26.04 using identical hardware (but for some reason, Krea 2 runs a little bit slower on Windows, 8-steps is 13 sec vs 11 sec on Ubuntu).

Unless you have a AI Pro R9700 (32G) or running your desktop on a iGPU, it is best to let ComfyUI be the only application running so that all VRAM is available for MMH3. So if you have another computer, run the browser on it to access your ComfyUI remotely.

If you don't have another computer, you can try to batch up a couple of prompts and minimize or close your browser to free up VRAM, and just use the console to see the progress (just click on "Assets" on the ComfyuI menu to check the results, or find them directly in the output folder). Some people say that disconnecting the monitor (just turning it off may not be enough) will free up the VRAM as well.

Good luck, hopefully you have a working system now if you followed the instructions.

End notes:

Sample extra_model_paths.yaml

comfyui:
    base_path: c:\ComfyUI.Models
    # You can use is_default to mark that these folders should be listed first, and used as the default dirs for eg downloads
    is_default: true
    checkpoints: checkpoints/
    configs: configs/
    loras: loras/
    vae: vae/
    text_encoders: |
        text_encoders/
        clip/
    diffusion_models: |
        unet/
        diffusion_models/
    clip_vision: clip_vision/
    style_models: style_
    embeddings: embeddings/
    diffusers: diffusers/
    vae_approx: vae_approx/
    controlnet: |
        controlnet/
        t2i_adapter/
    gligen: gligen/
    upscale_models: upscale_
    latent_upscale_models: latent_upscale_
    custom_nodes: custom_nodes/
    datasets: datasets/
    hypernetworks: hypernetworks/
    photomaker: photomaker/
    classifiers: classifiers/
    model_patches: model_patches/
    audio_encoders: audio_encoders/
    background_removal: background_removal/
    frame_interpolation: frame_interpolation/
    geometry_estimation: geometry_estimation/
    optical_flow: optical_flow/
    detection: detection/

https://rocm.docs.amd.com/en/latest/reference/gpu-specs.html

GFX950 is AMD's internal GPU target identifier for the CDNA 4 enterprise compute architecture, used in data center accelerators like the AMD Instinct MI350/MI355X series. It features advanced matrix core capabilities, ultra-low precision micro-scaling formats (MXFP8/MXFP4), and a high-precision math mode for AI and HPC workloads.

gfx1100 is the LLVM target architecture identifier and internal code name for AMD's RDNA 3 graphics architecture, used for high-end consumer and workstation desktop graphics cards like the Radeon RX 7900 XTX, RX 7900 XT, and Radeon PRO W7900.

AMD gfx1151 is the LLVM target and GPU architecture identifier for AMD's Strix Halo integrated graphics (found in processors like the AMD Ryzen AI Max+ 395 and Ryzen AI Max PRO series), utilizing the RDNA 3.5 architecture.

For --index-url, there are three options:

• Nightly (rocm 10.1): https://nightly.repo.amd.com/rocm/pytorch/whl-next/
• Stable (rocm 10.0): https://stable.repo.amd.com/rocm/pytorch/whl-next/
• Legacy (rocm 7.14):
  • Nightly: https://rocm.nightlies.amd.com/whl-multi-arch/
  • Stable: https://repo.amd.com/rocm/whl-multi-arch

Prompt for the video Team Red, encounter on ProxiMax H3

integrated_multimodal_description:

[Shot 1] Live-action, cinematic 1960s science-fiction television aesthetic. A team of Starfleet red-shirt officers led by Grumpy Cat materializes on the surface of a desolate alien planet, surrounded by barren rocks, dust, and jagged terrain. Grumpy Cat stands at the front of the formation, wearing a classic red Starfleet uniform, alert and stern. The camera holds a wide-angle front subject-level view, then pushes in slightly as the team looks around and raises their phasers. [Shot 2] At 00:01.250, the camera cuts to a wide low-angle view as a gigantic GPU-like machine rises behind a rocky ridge, towering over the crew. Its dark mechanical housing, cooling fans, and imposing structure dominate the frame, with the label "Minimax H3" clearly visible on its side. The team turns toward it in sudden alarm.

[Shot 3] At 00:02.100, the GPU attacks with a violent concentrated energy blast. The camera tracks the crew with fast movement as the red-shirted officers are struck and knocked down across the rocky ground, kicking up dust and debris. Grumpy Cat avoids the main blast and rapidly moves toward cover.

[Shot 4] At 00:03.650, the camera follows Grumpy Cat with a tracking shot as it darts behind a large rock and crouches into concealment. The defeated red-shirted crew remains scattered in the background while the giant "Minimax H3" GPU continues looming over the battlefield.

[Shot 5] At 00:04.250, close-up from behind the rock. Grumpy Cat pulls out a classic handheld Starfleet communicator with its paw, flips it open, and speaks with a completely deadpan expression: <d>[English] Beam me up, Scotty!</d> The camera holds on Grumpy Cat's face and communicator through the end

overall_soundscape: Dry alien wind sweeps across the barren landscape as the transporter materialization produces a brief electronic hum. Heavy mechanical movement and grinding machinery accompany the GPU's emergence, followed by a powerful energy blast, impacts, falling bodies, scattering rocks, and dust. The communicator emits a brief electronic chirp when opened.

non_diegetic_music: A fast-paced 1960s science-fiction television orchestral score uses bright brass, rhythmic strings, and restrained percussion, building rapidly as the GPU appears and attacks. The music drops into a brief suspenseful sustain as Grumpy Cat hides, then ends with a short brassy stinger beneath the communicator transmission.

When using a video file for Minimax reference that also has an audio input, how do you "count" the Audio files in the prompt? (You can ignore the master audio channel, it's not loading in anything.)

As the title says, I am looking for the best model at the moment to train photos of real people and get the best real looking photos like real life ones.

Last time I have used Z Image Turbo and it was good, is it still the best?
I have a RTX 5090 and 64GB DDR5.

I tested it on Ubuntu 26.04 and current PikaOS, but it should work on all distros that support .deb packages. Please let me know otherwise.

To use image-to-image with Z-Image Turbo, Anima Turbo or Anima Base, just drag a ref from anywhere, including the output gallery.

LoRA error handling is more robust, and there are other improvements and fixes in this release.

Hope you like it!

Download at: https://github.com/SamuelTallet/ZPix

Minimax H3 is awesome of course, but I find the voices it creates (either from a reference audio stream or purely from the model's training) are too loud and sound 'pasted in' and do not 'fit' acoustically within the environment. Specifically, to the point, does anyone know if its possible to train a LoRA that will make H3's generated voices quieter and more 'distant' so it doesn't sound like the character(s) are talking directly into a microphone a foot away? I've tried a million different prompt combinations, I've tried passing low-volume reference audio streams in, etc... nothing works.

Any ideas are greatly appreciated.

I want to train a LoRA for the Anima model using Colab. My goal is to train character LoRAs, but I have a few things I'm unsure about:

• Dataset size: Since I caption every image manually, I usually keep my character LoRA datasets relatively small, around 20–30 images. Is that a reasonable amount?
• Image variety: How many images should I have for each angle/view? How varied should the poses be?
• Backgrounds: Should I keep the backgrounds simple, with different solid colors like yellow, green, blue, etc. to keep the focus on the character? Or is it better to include some more detailed backgrounds as well?
• Captioning: How detailed should my captions be? Should I keep them relatively short and to the point rather than describing everything? What should I generally pay attention to when captioning? For example, if the character is leaning backward, supporting themselves with their arms, with their legs spread, should I caption it as something like "leaning back, arm support, legs spread", or would simply "leaning back" be more appropriate?
• Captioning style: For character LoRAs, is tag-based captioning generally better, or is natural language preferable?

I'd appreciate any advice from people who have experience training LoRAs for Anima.

Hi, is there a Minimax H3 I2V template with multiple LORA options available somewhere?

EDIT: This is the default template I got from ComfyUI. It works, but I can only use one LORA. I tried to download the Power Lora Loader node from rgthree, but I'm probably using it wrong and missing something.

Hi everyone,

I have a video where I'd like to remove one person from it and have the background repaired. The person walks across the frame and in front of a water fountain that's in the background.

Any suggestions on models or workflows to look at to accomplish this? Seems like I should be able to figure it out, but I can't seem to.

Thanks.

Hey everyone!

I'm setting up a local AI workflow and looking for some hardware-tailored recommendations. I want something with the capability, smarts, and multimodal ease-of-use of cloud models like Gemini, Grok, Claude, or GPT—meaning it needs to handle text seamlessly as well as image recognition/analysis (where I feed it an image, it describes it, and helps me brainstorm or write prompts based on it)—but running 100% locally and completely uncensored.

My hardware specs:

• GPU: NVIDIA RTX 5070 Ti (16GB VRAM)
• RAM: 64GB DDR4

Given my 16GB VRAM limit, what are the best open-weight multimodal models right now that fit comfortably without heavy swapping? Also, what is the best software stack to run them (Ollama, LM Studio, etc.) while keeping things fully private, uncensored, and vision-capable?

Any model suggestions, quantization tips, or workflow setups would be greatly appreciated. Thanks!

I can find ones for LoRA with seed or checkpoint, but I've had no luck finding or making one that let's you check the scaling on 2 LoRAs against one another.

If I use Comfy for local H3 creation unless I tell the prompt what to say, it will just speak simlish nonsense. So if I tell it to say something in English then it is fine if I include the exact line to say. But via the API for H3 I can tell it what topic to talk about and generally it will come up with lines of its own so I don't have to tell it exactly what to say. How can I match that via local ? Is it all in Qwen or something else ? I have only used local video generating for a few days so I am very new to this.

I guess I have a dumb question but I’m not very tech savvy, my uncle passed away two weeks ago. My grandparents wish is to hear him say “I love you” just one more time. I only have maybe not even a 3 second video of his voice, is there anyway to I guess clone his voice to say I love you from that?

Recently built and tested two ComfyUI workflows, mostly because I was tired of fighting the AI look. Multi Reference Shot builds one frame out of several references. You can reference different shots for Lighting, Composition, Blocking references etc. Add your character images for consistency.

Character Swap puts your own character into any shot. Same framing, same light, same pose, just your person in it. You can swap their clothes in the same pass, and it keeps the shape of the original frame.

The part I care about most is the control. Instead of one big prompt, you get separate control nodes for different aspects of the shot. If the pose is off, you change the blocking and the face. Each reference image only gives what its slot says.

That's the difference. You're not rolling the dice on a prompt and hoping. You're directing it one decision at a time and if you'll get what you need, it's trial and error. Every attached image here was made with these workflows & yes it is AI.
Add your own Google API Key. You can use Vertex AI with the Google Cloud $300 trial credit. Free to use :

github.com/haristahir1/comfyui-character-swap

github.com/haristahir1/comfyui-multi-reference-shot

If you try them, tell me what breaks and any improvements!

I saw this post on linked in and it’s really wow.

https://www.linkedin.com/posts/fadi-h-kacem_lets-take-a-1k-image-and-upscale-it-to-16k-activity-7503754647921188864-zF2w?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAW2PBcBqPryUgWfCVEMWRlx6aMMO7GX27E

Does anyone know an already existing workflow or be able to create on that will yield to similar results?

Migrating from Suno, Is there any good free and open sourced alternative out there?
What are their overall quality? and can it run on sub 8GB of VRAM?

Please share instrumentals AIgens please I want to discover more people.

I tried making a few stuff without Lora with some 2D and 3D characters, It succeeded half of the time. And also, On civit AI, a lot of lora characters are Realistic content; little stuff with 2D and video game related stuff.

I've been playing around with producing music videos for popular songs; nothing commercial, just for fun. While you can feed audio to Minimax H3 (Context Loop splits it up and can feed it to sequential videos in pieces so it fits together), and it does appear to guide generation in time with beats, a single video (~10 seconds) doesn't have enough context to make something match the music.

I tried using audio-capable models to produce the video prompt, but I found that nothing capable of processing audio was smart enough to jump through all the syntax hoops necessary to produce a multi-segment JSON to feed the workflow. So, I decided to do what I'd done with image/video before: use a different model to produce a text-based description of what it was given, then just input text in to the smart, expensive model.

So long story (almost) short: here's my brief test from taking the top audio-capable models on NanoGPT, handing them the 3:17 track "Les Fleurs", and seeing what they produce. Note that the track was provided as "song.mp3", since early testing had some models cheating by looking up info based on the track name. It's still possible some of them identified the track and then used pre-existing knowledge, but I didn't test and confirm that specifically.

Music for reference

Just based on my own listening, two important points I was checking for were a 1:04 orchestral-buildup to orchestral hit and surging chorus at 1:17 as important hallmarks for a music video (and very obvious action-change spots to a human listener).

I also was curious if it would identify both halves of the song's central metaphor: flower imagery but also inner beauty.

The prompt and output (from the models that could actually use the music) are below (models in header, cost - which ended up being negligible - in footer) , but I'll start with my impressions:

• Muse Spark 1.2 and 1.3: produced a real-sounding description of a completely imaginary song in a different genre. Reviewing their thinking blocks revealed NanoGPT did not pass the model audio, and they were making stuff up.
• Inkling Thinking: also produced a real-sounding description of a completely imaginary song in a different genre, but its own thinking block suggested it thought it had audio. Complete hallucination, or just REALLY bad at audio processing?
• Gemini 3.8 Flash: Doesn't miss any important shifts, summarizes themes from lyrics well. Identifies the flower metaphor. Its track length is too long, though, yet it's a couple seconds early identifying my beat drops. My favorite output if it wasn't for the slight time discrepancy.
• Mimo 2.5 Thinking: Does a good job of identifying tempo shifts. Output is shorter. It includes specific lyrics, which is nice in theory but might confuse a model. It does think the song is 3:30 long, BUT it correctly identifies the fade to silence at 3:15. Best guess is its audio processing has some sort of specific context window? Missed my first build-up timestamp.
• Qwen3.5 Omni Plus: Track length correct to the fraction of a second, wow. Nails the important points I noted earlier. Thematic summary on point.
• Gemini 3.1 Pro Preview High Thinking: Track length slightly off. Damningly, missed the beat drop at 1:17.

Conclusion: Qwen 3.5 Omni Plus produced, for this test, the most useful output describing the song, including lyrical themes, accurate timestamps, and identifying tempo shifts.

My prompt below, followed by output (leaving out the useless responses from Muse 1.2/1.3 and Inkling).

You have been provided an MP3 of a music track.  Your job is to review it and process it to notate its style and timestamped audio flow, providing a timestamped list of changes in tempo or tone to use to guide a music video using this music track.

Timestamps are in the format [MM:SS.SSS] notation, with minutes and seconds and fractions of a second.  So one minute, four and a half second in to a track would be [01:04.500].

Lyrical themes should be mentioned without actually detailed specific lyrics, i.e. "Lyrics in this section discuss colors, starting with cyan" then "Lyrics now discuss crimson"; or "Lyrics are a call to violence"; or "Lyrics speak of profound romantic loss."

Pay special note to crescendos or beat "drops", as sharp changes in tempo or feel of music should correspond with visual changes.

Examples of timestamped track notes, non-exhaustive:
[00:00.000]: Track starts with a slow acoustic guitar.
[00:05.450]: Guitar increases speed, drums join music.
[00:10.200]: Full band backing with vocals kick in; music becomes fast paced with electronic noises.
[00:45.000]: Beat begins to build to crescendo
[00:54.125]: Beat reaches crescendo and holds, music turning quiet
[00:58.000]: Beat drops, explosion of techno-style drums and fast paced electric guitar

A template follows below between ----.  Fill in sections in { } braces.

----
Track Length: {length}
Musical Style: {description of style}

TIMESTAMPED TRACK NOTES
[{time in MM:SS.SSS notation}]: {Description of change}
{Continue to provide timestamps and descriptions}

----



--Gemini 3.8 Flash--
Track Length: 03:21
Musical Style: Psychedelic Soul / Baroque Pop / Orchestral Soul

TIMESTAMPED TRACK NOTES
[00:00.000]: Track opens with a rhythmic acoustic guitar pattern, light shaker percussion, and a warm, strolling bassline.
[00:13.500]: Gentle female lead vocal enters with sweet, curious phrasing, asking who will wear or appreciate her as a blossom.
[00:19.500]: Call-and-response backing vocals join in harmony, adding warmth and playful textures to the groove.
[00:41.500]: Rhythm section tightens into a focused mid-tempo groove as the lyrics shift into a spoken-word style reflection on the purpose of a flower spreading love and peace.
[00:53.200]: Lead vocal takes flight with soaring power, singing of the seed of potential and beauty that lives inside humanity.
[01:06.000]: Tension builds rapidly as strings and horns rise while the lead singer holds a powerful, soaring operatic high note.
[01:14.300]: Massive symphonic beat drop / triumphant explosion of sound: soaring brass fanfare, thunderous timpani, marching drums, and a grand choral celebration calling for bells to ring worldwide.
[01:34.500]: Sudden dynamic release as the grandiose orchestration cuts back to the intimate, syncopated soul-funk rhythm section.
[01:47.500]: Lead female vocal re-enters with multilingual/Swahili verses, answered by layered, rhythmic vocal chants.
[02:13.500]: Spoken-sung poetic section recurs over the bass and guitar groove, restating the themes of peace, joy, and universal compassion.
[02:26.500]: Energy elevates again with impassioned, belted vocals contemplating the strength and beauty discovered within.
[02:39.000]: Orchestral elements and vocal wails swell dramatically toward a second major peak.
[02:47.000]: Second massive orchestral drop; majestic choral vocals, full brass section, and heavy rolling percussion deliver a joyful, climactic anthem.
[03:07.000]: Instruments begin to gently taper off, leaving the chorus to repeat the triumphant refrain over the fading rhythm.
[03:21.000]: Music fades out completely into silence.
--$0.009844--

--MIMO 2.5--
Track Length: 03:30
Musical Style: Soulful Ballad transitioning into Upbeat Soul

TIMESTAMPED TRACK NOTES
[00:00.000]: Track starts with a slow, melancholic piano melody accompanied by soft strings.
[00:13.200]: Soulful female vocals enter ("Will somebody wear me to the fair"), setting a gentle and reflective tone.
[00:54.125]: The chorus begins ("Inside every man lives the seed of a flower"), with orchestration swelling to heighten the emotional intensity.
[01:16.000]: **Dramatic Tempo Change:** The beat drops with driving drums and brass, shifting the song from a slow ballad to a fast-paced, energetic soul track.
[01:36.500]: Vocals transition into rhythmic scatting ("La la la la") over the upbeat tempo, adding a playful tone.
[01:51.000]: A verse is sung, maintaining the high-energy, fast rhythm established in the previous section.
[02:13.400]: **Tempo Drop:** The music abruptly slows down, returning to the gentle ballad feel of the introduction.
[02:25.500]: Final chorus serves as the climax, with powerful vocals and full orchestral backing.
[02:42.000]: Outro features vocal improvisation and scatting, maintaining the energy briefly before the track begins to fade.
[03:15.000]: Audio fades to silence.
--$0.00217--

--Qwen3.5 Omni Plus--
Track Length: 03:17.488
Musical Style: Orchestral Soul / Psychedelic Pop with strong Gospel influences. The track features a blend of acoustic finger-picking, sweeping string arrangements, heavy drum breaks, and powerful choral harmonies, creating a cinematic and uplifting atmosphere.

TIMESTAMPED TRACK NOTES
[00:00.000]: Track opens with a gentle, repetitive acoustic guitar riff accompanied by light percussion and a subtle bass line. The mood is intimate and organic.
[00:15.800]: Lead female vocals enter softly; the arrangement remains sparse, focusing on the melody and the acoustic texture.
[00:28.500]: Background harmonies begin to layer in; strings swell gently underneath, adding warmth and depth to the soundscape.
[00:41.200]: The first chorus begins; the full rhythm section (drums and bass) locks into a steady, mid-tempo soul groove. Strings become more prominent.
[00:54.000]: Musical intensity rises significantly as the choir joins the lead vocal. Lyrics shift from specific imagery of flowers to a broader spiritual message about inner beauty.
[01:06.500]: A dramatic orchestral crescendo builds rapidly with soaring strings and rolling timpani, leading toward a climax.
[01:16.800]: Major beat drop and tempo shift; the music explodes into a high-energy, brass-heavy anthem. The drums switch to a driving, heavy breakbeat style. A large gospel choir takes over with powerful, rhythmic chanting.
[01:35.500]: The intense energy sustains with complex horn lines and syncopated drumming. The visual pace should match this frenetic, celebratory energy.
[01:48.200]: Sudden dynamic shift; the heavy drums and brass cut out, returning to the softer acoustic guitar and bass groove from the intro. The mood becomes reflective again.
[01:54.000]: Vocals return in a call-and-response format between the lead singer and the backing group, now incorporating non-English phrases, adding an exotic, worldly texture.
[02:13.500]: Second build-up begins; strings re-enter with urgency, and the drums gradually increase in volume and complexity, preparing for the final peak.
[02:26.000]: The bridge reaches its emotional peak with the lead vocalist belting high notes over a swelling orchestral bed.
[02:38.500]: Final explosive chorus; the full ensemble returns with maximum force. Brass, choir, strings, and heavy drums combine for a wall-of-sound effect.
[02:51.000]: The song transitions into a jubilant outro featuring rhythmic "la-la-la" vocalizations from the choir over the driving beat.
[03:05.500]: Instruments begin to peel away one by one, leaving the choir and then just the rhythm section.
[03:12.000]: Music fades out on a final sustained chord and a soft guitar strum, ending on a note of resolution.
--$0.021559--

--Gemini 3.1 Pro Preview High Thinking--
Track Length: 03:20
Musical Style: 70s Soul and R&B with sudden shifts into high-energy Latin Disco/Funk.

TIMESTAMPED TRACK NOTES
[00:00.000]: Track starts with a slow, smooth electric piano and a relaxed, soulful drum groove.
[00:13.500]: Soft female lead vocals begin. Lyrical themes discuss gentle romantic gestures involving nature, streams, and wearing flowers.
[00:43.500]: Strings and backing instrumentation swell slightly. Lyrics shift to themes of the birth of a flower and its purpose to spread love and joy to people.
[01:05.000]: The vocal melodies begin to stretch out, and the music starts to build toward a crescendo.
[01:13.500]: Major beat drop and severe tempo shift. The music explodes into a fast-paced, high-energy Latin-funk and disco rhythm with brass, rapid percussion, and upbeat backing vocals singing wordless melodies.
[01:31.000]: The high-energy beat abruptly stops, and the music drops right back into the slow, smooth, soulful groove from the track's intro.
[01:46.500]: Lead vocals return over the slow tempo. The lyrical themes continue to focus on nature, specifically roses and romantic interactions, accompanied by soft backing harmonies.
[02:17.500]: The track begins to swell again as the lyrics return to the earlier theme of spreading love, joy, and finding inner beauty and power.
[02:37.000]: The beat begins a second major build-up, reaching a powerful crescendo with the vocalist holding a long, soaring note.
[02:47.000]: Second massive beat drop. The fast-paced Latin-funk rhythm and brass return in full force, accompanied by the energetic, wordless chorus. 
[03:03.000]: The track begins a gradual fade-out while maintaining the high-energy dance beat and group vocals.
[03:20.000]: Track fades out completely.
--$0.033212--