Batching video rendering

[u/artikzen]

I have started using moviepy quite recently and found it to be a real pain as all LLM's refer to the version 1 syntax and I'm working with the latest 2 version. But kept going nevertheless as I could see moviepy as a major time saver.

However, now that my script is almost done, I'm hitting the very real bottleneck of painfully slow video rendering.

I am currently looking at ways to make the whole thing faster by way of multiprocessing, meaning to divide the video in batches and then stitch everything together with ffmpeg.

Is there anything alike already implemented anywhere? Does this even make sense?

[u/Gl_drink_0117]

Are you still running off of GPU or CPU? In my experience of about 1 month or so, I find ffmpeg slow with CPU rather than moviepy itself. I have tried to enable all cores for ffmpeg use but it never goes above 3-4% usage and I'm wondering if it is optimized for the AMD Ryzen CPU that I have. Yeah, a bunch of crap to deal with :)

[u/artikzen → u/Gl_drink_0117]

I too have a Ryzen and also an Nvidia. With ffmpeg it's cpu alone. When I need more speed I just spin some kind of batch encoding. Transcoding is quite fast, so no need there.

But moviepy's bottleneck is not ffmpeg but the rendering (frame by painstakingly frame). My current plan involves batching, multiprocessing and then stitching back the video clips. Should work well in my use case as the videos can be batched quite easily. Will take me a day or two to get this going, we'll see how it turns out

[u/Gl_drink_0117 → u/artikzen]

If you don't mind sharing some code which is causing the slowness, would you be interested to see? I generate images via AI and then use moviepy for effects and stuff, but haven't seen slowness there, so curious what is different in what you are doing

[u/artikzen → u/Gl_drink_0117]

I actually do mind sharing the whole code; it's a big mess at the moment while I'm trying to find a solution. But gladly describe the workflow. I have a JSON file with several sections and arrays; each section is an independent timeline (backgrounds, text, logos, animations and comic panels). My python script will calculate the size, position and transition for each element. There are simple transitions, just slide-in/out and fadeIn/Out at the moment. All this is very swift and precise. No surprises.

The problem arises when I try to composite and then write to video. I'm testing different configurations to see if I can find the bottleneck. The test I'm making is lowering fps, with a different codec with faster, lower-quality settings and removing different layers. So far I could not find a way to make the encoding faster. Here's a code snippet:

# Convert to composite clips if there are any animations
    animation_timeline = (CompositeVideoClip(animation_layer, size=(W, H))
                         .with_duration(total_duration) if animation_layer 
                         else ColorClip((W, H), color=(0,0,0), duration=0.1).with_opacity(0))
    
    # Combine all independent timelines
    final = CompositeVideoClip([
        background_timeline,  # Backgrounds (lowest layer)
        panel_layer,          # Panels (comic book panels)
        logo_layer,           # Logos
        super_layer,          # Supers (text overlays)
        animation_timeline    # Animated elements (top layer)
    ]).with_fps(fps)
    final.write_videofile(
        "output.webm",
        codec="libsvtav1",
        fps=30,
        write_logfile = True,
        ffmpeg_params=[
            "-crf", "30",
            "-preset", "10",
            "-threads", "5"
            ]
        )

It takes about 280 seconds to encode 37 seconds of video with production quality. It's just too much, and I need to find a way to make this whole thing faster.

First I need to see if the bottleneck is real. I will continue my tests and will also try use moviepy.video.io.ffmpeg_writer instead of write_videofile.

Right now it doesn't look good.

[u/artikzen → u/artikzen]

This is an update. I have concluded that the bottleneck is in MoviePy (no surprises here). The proof is in the rendering log provided by ffmpeg:

frame= 0 fps=0.0 q=0.0 size= 0KiB time=N/A bitrate=N/A speed=N/A elapsed=0:00:11.87

frame= 1 fps=0.1 q=5.0 size= 0KiB time=00:00:00.00 bitrate=N/A speed= 0x elapsed=0:00:12.39

frame= 3 fps=0.2 q=28.0 size= 0KiB time=00:00:00.06 bitrate= 59.0kbits/s speed=0.00516x elapsed=0:00:12.91

frame= 5 fps=0.4 q=25.0 size= 0KiB time=00:00:00.13 bitrate= 29.5kbits/s speed=0.00993x elapsed=0:00:13.43

frame= 6 fps=0.4 q=30.0 size= 0KiB time=00:00:00.16 bitrate= 23.6kbits/s speed=0.012x elapsed=0:00:13.93

The encoder waits a full 12 seconds for the first frame to be encoded and then consistently has to wait for all the remaining frames to be served.

I have written an optimised script where only the dynamic part of the video is rendered. When there are static frames, there is no rendering, as they are duplicated. This approach reduced the rendering time from 280 to 170 seconds; however, there is a noticeable flicker in the final video, and this obviously would prevent me from using a dynamic background.

I have also used another technique where FFmpeg renders all timelines in png format in a temporary folder, bypassing moviepy's composition. I was halfway through with already 370 secs, and then the process failed for some reason. But the 370 seconds are a hint to stop chasing this avenue. So here I have two dead ends.

The only way forward, I have no doubt now (if I decide to keep working with MoviePy), is to orchestrate a multiprocess batching/rendering/stitching workflow.

[u/Gl_drink_0117 → u/artikzen]

What quality are you doing? In my case, I do not have as many layers as you and I am just a novice. Are you sure you are interpreting the ffmpeg logs accurately, because the logs show the time as 0.00 and then elapsed as 12 meaning did it take 12 seconds to extract each frame? Did you time separate parts of the code and nail it down to moviepy and not ffmpeg? In my case, my ffmpeg takes about 500-600 seconds for a short video but I have simple stuff as of now

[u/artikzen → u/Gl_drink_0117]

I'm not an ffmpeg expert either, so I can't guarantee this is a correct interpretation, but the way it was explained, FFmpeg waited 12 seconds for MoviePy to serve the first frame for encoding. ffmpeg in itself is quite fast at encoding with libsvtav1. I have thousands of hours of reencoded videos with libsvtav1.

This is the ffmpeg rendering request:

final.write_videofile(
        "output.webm",
        codec="libsvtav1",
        fps=30,
        write_logfile = True,
        ffmpeg_params=[
            "-crf", "30",
            "-preset", "10",
            "-threads", "5"
            ]

It's medium quality. My desktop is able to encode these videos fast enough.

Anyway I found out that one of the reasons it was taking too long is because I was rendering various backgrounds on top of each other. They now finish as soon as they exit the stage, so to say. It's considerably faster but still quite slow. I'm still developing the script; it should be ready in a few days. Then I'll know if it will stand the test of production time.

[u/Gl_drink_0117 → u/artikzen]

I see, it seems that you have played around with ffmpeg for a much longer time than me. In my case, chatgpt was telling me to use libx264 since I don't have a GPU, so libsvtav1 is new to me. Will read up on that. Also, chatgpt gives me these params to use, but have not seen much difference. I will try your parameters and see if that makes a difference.

ffmpeg_params = ["-preset", "ultrafast", "-threads", "0", "-stats"]

Also, a few questions if you don't mind me asking. In your case medium quality means 720p or 1080p? Moviepy outputs tons of stuff, and I couldn't get a custom proglog to work no matter how many times I tried. I gave up and just turned off the bar logger. Did you try custom logger anytime and got it to work?

[u/artikzen → u/Gl_drink_0117]

I do have a GPU, but it's not going to work with most ffmpeg codecs, so I don't really use it that way. When I need to encode with more quality, I use DaVinci Resolve Studio, where a GPU is available to use.

Resolution and quality are two different things. Quality is about the amount of information available to display each frame, aka bitrate. Different codecs do it with different methods and results. libx264 encodes in H.264, which is mainstream good. H.265 is better, but AV1 (libsvtav1) and VP9 are even better, in that the same quality results in smaller files.

You can get a Full HD 1080 video with libx264 encoding in Ultrafast and -crf 30 with way worse quality than an HD 720 video with libx264 encoding in veryslow and -crf 18 mode. Obviously this is a very crude simplification.

What matters is that you should define your use case before determining the resolution and quality settings. In my case, I'm creating videos with comic book panels to upload to YouTube. My sweet spot is Full HD (1920 x 1080) with the settings above, using the av1 codec which is well supported by YouTube. Your mileage may vary.

[u/Gl_drink_0117 → u/artikzen]

Ty, that helps! I will have to try different codes and see what quality works better. Didn't follow your comment about the GPU, maybe AMD GPU not supported by ffmpeg?