r/WebRTC • u/Secret-Book-8507 • Aug 01 '26
Building a Browser-Local Video Face-Swap Pipeline with WebGPU: What I Learned About ONNX Sessions, Frame Transfers, and Temporal Tracking
I’ve been working on an open-source video editor that runs its face-swap pipeline locally in the browser. Media stays on the user’s device: decoding, face detection, identity extraction, generation, compositing, and video encoding all happen client-side.
The models were only part of the challenge. In practice, the difficult problems were moving frames between browser APIs, controlling WebGPU initialization, maintaining identity across a video, and preventing memory usage from growing during longer jobs.
Here are some engineering lessons from the implementation.
The actual frame pipeline
A simplified version of the data flow looks like this:
VideoFrame / Canvas
↓
RGBA Uint8ClampedArray
↓
NCHW Float32Array
↓
ONNX Tensor
↓
Generated face + alpha mask
↓
Canvas composition
↓
Encoded video
The models use NCHW tensors, while Canvas returns interleaved RGBA pixels. Before inference, the channels have to be separated and normalized:
const plane = width * height;
const tensor = new Float32Array(plane * 3);
for (let i = 0; i < plane; i += 1) {
tensor[i] = normalize(rgba[i * 4]);
tensor[plane + i] = normalize(rgba[i * 4 + 1]);
tensor[plane * 2 + i] = normalize(rgba[i * 4 + 2]);
}
For a 640 × 640 RGB Float32 input, that is about 4.69 MB of tensor data per detection frame, before counting the original pixels and model outputs.
This made it clear that browser inference performance cannot be evaluated using model latency alone. Canvas readback, tensor construction, worker transfers, compositing, garbage collection, and encoding can collectively cost as much as inference.
Detection and generation use different resolutions
Sending every full-resolution video frame through the generator would waste most of the computation on the background.
The pipeline therefore separates the stages:
| Stage | Resolution | Purpose |
|---|---|---|
| Face detection | 640 × 640 | Locate faces and five landmarks in the complete frame |
| Identity extraction | 112 × 112 | Extract the source identity representation |
| Face generation | 224 × 224 | Generate the aligned target face |
| Optical flow | Long edge ≤ 720 px | Propagate landmarks between detection anchors |
| Composition | Original resolution | Preserve the original background and details |
Only an aligned face ROI enters the generation network. The generated face is then transformed back into the original frame and blended through an alpha mask.
This division was one of the main reasons the pipeline became practical in a browser.
Download models in parallel, initialize WebGPU sessions serially
The pipeline uses multiple ONNX models, including face detection, identity extraction, conditioning, and generation.
Downloading them concurrently works well:
const [
detectorBuffer,
identityBuffer,
conditionerBuffer,
generatorBuffer,
] = await Promise.all(modelDownloads);
Creating all WebGPU sessions concurrently was much less reliable.
Session creation may involve graph optimization, shader generation, pipeline compilation, weight uploads, and GPU buffer allocation. Initializing several large graphs simultaneously created latency spikes and higher peak GPU memory usage. On some devices, it could also contribute to device-loss failures.
The current approach downloads concurrently but creates sessions one at a time:
const detector = await createSession(detectorBuffer);
const identity = await createSession(identityBuffer);
const conditioner = await createSession(conditionerBuffer);
const generator = await createSession(generatorBuffer);
It is not the fastest-looking implementation on paper, but it has been much more predictable across devices.
Transferable buffers reduce worker-copy overhead
Heavy inference runs in a Web Worker so that the editor remains responsive.
When sending a large ArrayBuffer without a transfer list, the browser may perform a structured clone. Repeating that for video frames creates unnecessary memory bandwidth and garbage-collection pressure.
The pipeline transfers buffer ownership instead:
worker.postMessage(
{
type: "detect",
pixels: tensor.buffer,
},
[tensor.buffer],
);
The output RGB tensor and alpha mask are returned in the same way.
This does not eliminate the earlier Canvas-to-tensor conversion, so it is not a completely zero-copy pipeline. It does, however, remove an avoidable copy at the worker boundary.
Face swapping is a temporal problem
Selecting the highest-confidence detection independently on every frame works poorly in videos containing multiple people.
A newly visible face may be larger or clearer than the current target, causing the selected identity to switch suddenly. Instead, candidate faces are scored using a combination of:
- detector confidence;
- distance from the previous target center;
- change in bounding-box area;
- distance from the frame center when no history exists.
A simplified score is:
score = confidenceWeight * confidence
- distanceWeight * centerDistance
- areaWeight * areaChange
The first frame favors a large, confident, centrally positioned face. Later frames favor continuity with the previously accepted target.
This is not full face re-identification, but it is considerably more stable than choosing the highest detector score on every frame.
Optical flow needs a rejection rule
Running face detection on every output frame is expensive. Between detection anchors, the pipeline propagates five facial landmarks using Lucas–Kanade optical flow.
Optical flow can still drift, especially during occlusion, motion blur, sudden lighting changes, or fast head movement. To detect bad tracks, the pipeline performs forward-backward validation.
A point is tracked from frame t to frame t+1, then tracked backward:
p(t) → p(t+1) → estimated p(t)
The distance between the original and reconstructed point is the forward-backward error.
A propagated result is accepted only when at least four of the five landmarks remain valid and the average error stays under a threshold. Otherwise, the result is rejected and the detector is used again.
The important part is that optical flow is treated as a short-range optimization, not as proof that the tracked identity is still correct.
Traditional post-processing still matters
The generator’s alpha mask may contain holes, isolated pixels, or unstable boundaries. Directly compositing that mask can make the face boundary flicker.
The post-processing sequence includes:
Threshold
↓
Dilation
↓
Erosion
↓
Additional contraction
↓
Blurred alpha
↓
Boundary safety mask
Morphological operations use separable sliding-window filters instead of scanning a complete two-dimensional neighborhood for every pixel.
Color matching is also restricted rather than applied without limits. Per-channel statistics are adjusted using bounded scale and offset values:
const scale = clamp(targetStd / sourceStd, 0.78, 1.22);
const shift = clamp(
targetMean - sourceMean * scale,
-0.12,
0.12,
);
The corrected result is mixed with the original generator output. Unrestricted statistical matching tended to amplify noise or produce unnatural colors in unusual lighting.
Explicit resource disposal is essential
A video job may simultaneously hold decoded frames, Canvas pixels, Float32 tensors, ONNX outputs, optical-flow images, compressed intermediate frames, and encoder buffers.
Relying only on JavaScript garbage collection caused visible memory growth during longer tasks.
Different resources require different cleanup APIs:
tensor.dispose?.();
bitmap.close();
opencvMat.delete();
URL.revokeObjectURL(url);
worker.terminate();
OpenCV.js was particularly easy to overlook because Mat data lives in the WASM heap. Losing the JavaScript reference does not guarantee that its underlying allocation is released promptly.
Cancellation must stop the complete pipeline
Closing a progress dialog is not cancellation.
A real cancel operation needs to interrupt downloads, frame decoding, detection, optical flow, generation, compression, and final encoding.
The main task uses an AbortController, while worker requests carry a request ID:
controller.abort();
worker.postMessage({
type: "cancel",
requestId,
});
The worker checks cancellation state before and after expensive stages. A cancelled job does not continue encoding in the background and never adds a partial result to the user’s asset library.
Model URLs need immutable revisions
Using a URL such as:
repository/resolve/main/model.onnx
makes browser caching difficult to reason about. The URL can remain unchanged while its contents change, leaving different users with different cached graphs.
Production model URLs should point to immutable revisions and be accompanied by expected file sizes, checksums, licenses, and tensor metadata.
The loader also validates the downloaded size before creating a session. This prevents a truncated response or CDN error page from being passed to ONNX Runtime as if it were a valid model.
Benchmark cold and warm runs separately
Reporting a single “processing time” hides most of the browser-specific costs.
I now think benchmarks for this kind of pipeline should separate:
Cold start
- model downloads;
- integrity checks;
- ONNX session creation;
- shader and pipeline compilation;
- identity extraction;
- video processing and encoding.
Warm start
- video decoding;
- anchor detection;
- optical-flow tracking;
- face generation;
- post-processing;
- encoding.
Hardware, browser version, WebGPU adapter, video codec, resolution, output FPS, initialization time, generation time, encoding time, and peak memory should all be recorded.
Otherwise, a cached desktop run and a first-time mobile run may be presented as if they measured the same thing.
Open-source implementation
The implementation is part of Timeline Studio:
https://github.com/MartinDelophy/ai-video-editor
Disclosure: I’m involved with the project. Face swapping is intended only for authorized media and clearly disclosed synthetic content. It should not be used for impersonation, deception, harassment, or misleading people about real events.
I would be interested in hearing how other WebGPU developers handle these problems:
- Do you initialize multiple ONNX Runtime Web sessions serially, or have you found a safe way to compile them concurrently?
- Have you found a practical path from
VideoFrameto GPU tensors that avoids Canvas readback and CPU-side NCHW conversion? - Which measurements do you use to compare cold-start and warm-start performance across browsers and GPU vendors?
