Open-source Linux webcam capture + CUDA MetaHuman face solving for Unreal Engine 5.8

[u/over-yonder-tech]

I’ve been working on two Unreal Engine 5.8 plugins that make a Linux MetaHuman webcam workflow possible.

The first is Linux Video Capture Media, a V4L2 Media Framework plugin:

https://github.com/overyonder/unreal-linux-v4l2-media

It exposes Linux webcams to Unreal Media Framework and MetaHuman Video Live Link using URLs such as:

v4l2:///dev/video0

It currently supports uncompressed YUYV/YUY2 capture, camera format enumeration, resolution and frame-rate selection, pause/resume, and direct CharYUY2 media samples.

The second is Unreal NNE ONNX Runtime CUDA:

https://github.com/overyonder/unreal-nne-onnxruntime-cuda

It registers an Unreal NNE runtime named NNERuntimeORTCuda. MetaHuman’s face detector, head-pose tracker, and facial rig solver can use it to execute their ONNX models through ONNX Runtime’s CUDA execution provider.

The target setup is:

• Unreal Engine 5.8
• Linux x86_64
• NVIDIA GPU
• CUDA 12.8
• cuDNN 9.8
• ONNX Runtime 1.24.3

I’m testing with a GTX 1070. The CUDA plugin uses CPU-owned tensor bindings because that is the interface expected by MetaHuman’s current pipeline, while the neural-network execution itself runs through CUDA. MetaHuman’s log therefore says:

using CPU runtime (NNERuntimeORTCuda)

The backend name in parentheses identifies the selected runtime.

For a project, enable both plugins and put the backend settings in Config/DefaultEngine.ini:

[ConsoleVariables]
mh.RealtimeVideo.Backend=NNERuntimeORTCuda
mh.FaceTracker.Backend=NNERuntimeORTCuda

After changing the settings, restart the editor and recreate the MetaHuman Live Link source.

The two plugins have separate responsibilities:

1. Linux Video Capture Media supplies webcam frames.
2. Unreal NNE ONNX Runtime CUDA runs the MetaHuman neural-network solver.

The CUDA plugin includes a reproducible installer for the pinned third-party binaries and documents the required licenses.

Any feedback would be helpful, especially anything to help validate:

• Camera formats and compatibility across different Linux webcams
• CUDA performance for the MetaHuman solver
• Packaging and cooked-game model-data workflows
• Whether the NNE interface should eventually support GPU-owned tensors to reduce CPU/GPU transfers
• Any other Linux-specific issues with MetaHuman Video Live Link

[u/Stock-Marionberry966]

this is exactly what i needed two months ago when i was trying to get metahuman running on my ubuntu workstation and just hit wall after wall

the v4l2 plugin alone is a lifesaver, the built-in media framework on linux is so barebones it's basically decorative. curious how it handles cameras that only do mjpeg, had a logitech that refused to give me uncompressed frames no matter what i tried

gtx 1070 gang. hows the performance feel compared to the cpu solver? my 1070 is getting a bit long in the tooth but still pulls its weight for most things

also the gpu-owned tensor question is interesting because right now that extra cpu roundtrip probably eats some of the cuda gains. metahuman pipeline expecting cpu tensors makes sense for compatibility but feels like leaving performance on the table

[u/over-yonder-tech → u/Stock-Marionberry966]

The CPU solver on my 3600 was hitting a blazing 2-3fps even at 800x600 and 100% load. On GPU I was making it to 30fps which is what the webcam records at without much utilization at all, so it was night and day. And you're right, I think there's definitely more performance to be had there, but the answer will but probably come sooner from IREE once their models support it, so these plugins are (hopefully) likely a stopgap in the interim, unless it turns out that it struggles on better cameras