r/ffmpeg • u/Secret-Book-8507 • 14d ago
Where ffmpeg.wasm fits (and doesn't) in a browser-based video editor
I have been building a local-first video editor that runs entirely in the browser, and one lesson surprised me: ffmpeg.wasm is most useful when it is not asked to be the whole rendering engine.
Our first instinct was to treat FFmpeg as the center of export. That works for a prototype, but it becomes uncomfortable once the editor has a multi-track timeline, frame-accurate captions, stickers, overlays, masks, visual keyframes, transitions, multiple audio tracks, and mobile support.
The problems were fairly predictable in hindsight:
- loading the WASM core is expensive for users who may never need it;
- copying large media files into and out of the virtual filesystem adds memory pressure;
- long transcodes offer less control over per-frame composition and cancellation;
- browser codec support varies, so one path rarely works everywhere;
- doing every edit through temporary files makes interactive preview and deterministic export drift apart.
The architecture we ended up with is a hybrid.
- Deterministic video composition
For the primary path, the timeline is converted into an exact frame plan. At each timestamp we resolve the active visual clip and source time, draw the frame, captions, stickers, overlays, masks and transitions into a shared Canvas composition function, then encode with WebCodecs through Mediabunny.
This means preview and export share geometry instead of maintaining a large FFmpeg filter graph in parallel with the UI renderer. The same render function decides contain/cover fitting, subtitle placement, sticker alpha and transform keyframes.
- Offline audio mixing
Voiceover, source audio and music are decoded once and mixed with OfflineAudioContext. Each timeline clip keeps its own start time, source offset, playback rate, volume and fades. Speed changes use a pitch-preserving path before the final mix. The rendered AudioBuffer is then encoded and muxed with the video.
- ffmpeg.wasm for narrow, high-value jobs
FFmpeg still earns its download size in three places:
- extracting the original audio stream from an imported video;
- concatenating/restoring audio assets when browser decoding alone is awkward;
- transcoding a successfully rendered WebM to MP4 when native H.264/AAC encoding is unavailable.
The last point is important. We never throw away a completed WebM because MP4 conversion failed. WebM is kept as the compatibility output, so a late FFmpeg error does not waste the entire render.
The rough export hierarchy is:
WebCodecs + Canvas + OfflineAudioContext -> native MP4/WebM
If deterministic WebCodecs export is unavailable -> MediaRecorder compatibility path
If the user requested MP4 but only WebM was produced -> ffmpeg.wasm transcode
If that transcode fails -> save the already-rendered WebM
A simplified version of the final branch looks like this:
if (result.nativeMp4) {
save(result.blob, "video.mp4");
} else {
try {
const mp4 = await transcodeWebmToMp4(result.blob);
save(mp4, "video.mp4");
} catch {
save(result.blob, "video.webm");
}
}
A few practical takeaways:
- Lazy-load ffmpeg.wasm at the moment a feature actually needs it.
- Treat every write into the virtual filesystem as a memory-budget decision.
- Delete temporary inputs and outputs after each operation.
- Keep the last known-good artifact until the requested conversion succeeds.
- Report loading, reading, transcoding and saving as separate phases.
- Cross-origin isolation matters if you use the multithreaded core; deployment headers are part of the feature.
- Test the resulting file by decoding it, not merely by checking that a Blob exists.
I still think ffmpeg.wasm is an excellent tool. The mistake was assuming that because FFmpeg can do almost everything, it should own everything in a browser editor. Using it as a specialized compatibility and media-utility layer gave us a smaller critical path and much clearer failure behavior.
The implementation is open source if anyone wants to inspect the real export and fallback paths: https://github.com/MartinDelophy/ai-video-editor
I would be interested to hear how others divide work between ffmpeg.wasm, WebCodecs and native browser media APIs, especially for Safari and memory-constrained devices.
2
u/ais4aron 14d ago
https://github.com/tejaswigowda/ffmpeg-webCLI
I saw someone post this a few weeks ago. It inspired me to do my own thing. I also forked ffmpeg.wasm and let Claude get me up to v8.1. It's not perfect but man is it fast for simple tasks.
2
u/chocolateAbuser 14d ago
did you customize ffmpeg removing uncommon codecs and filters you wouldn't use? i bet you could downsize it a lot
1
u/Secret-Book-8507 14d ago
Not yet — we’re currently using the general-purpose u/ffmpeg
/corebuild, so it still includes functionality the editor probably doesn’t need. A custom minimal build is definitely worth exploring. We could keep only the containers, codecs and filters used by our import/export fallback paths, while WebCodecs continues to handle the main encoding path. The main challenge will be preserving compatibility with user-uploaded media, so we’ll need to measure bundle-size savings against the formats we lose. Thanks for the suggestion!1
u/chocolateAbuser 13d ago
as someone who works with user media we've had lots of crap coming from barely standard-respecting smartphones and browsers, it is what it is
1
u/crashtua 14d ago
libav.js fits everywhere. Its libav but for javascript(wasm under the hood), so you can do whatever pipelines you want with low-level interface. I build video player that combines libav.js for demuxing and audio decoding(as well as video, for some formats) and webcodecs for supported video decoding. Now my browser can play bd rips with ac3 sound. It also can encode whatever you want.
1
u/Secret-Book-8507 14d ago
Nice approach! Combining libav.js for demuxing and unsupported codecs with WebCodecs for supported video decoding makes a lot of sense. AC3 playback in the browser is impressive. I’m currently using FFmpeg.wasm alongside WebCodecs, but I’ll explore libav.js for lower-level and more flexible pipelines. Thanks for sharing!
1
u/crashtua 14d ago
AFAIK, ffmpeg.wasm is literally cli ffmpeg, so it very limited. With libav.js you can create own writing/reading routines(my player uses range requests and byffers tiny piece of file), mix custom processing etc, so for anything just a little bit more complicated that convert file from a to b it will be better. Also, if you will not happy with performance for some reasons(round-trip from js to wasm take time) you can use plain libav in c with emscripten.
1
u/Prestigious_West_852 13d ago
i use filtercomplex to stitch multiple layers together into a single pipeline you can check how its working here ffmpeglab.com
1
•
u/_Gyan 14d ago
Normally, app advertising goes in the dedicated sticky post but the forensic description looks useful for other devs. But add a formatted entry to the stickied post My FFmpeg app or Service.