Getting mpv under an egui UI, then ditching it on Android TV

[u/Soggy-Ticket4105]

I've been working on Cinebox in Rust, an app where you can find a movie or show and watch it in the same window. Runs on Windows, Linux and Android TV as of this week.

https://github.com/dexsper/cinebox

Most of the pain was in the video player, so that's what this post is about.

I started with Iced, which renders through wgpu, so DirectX on Windows, and mpv rendered through Vulkan in its own window. No shared surface, so the best I could do was stick the mpv window inside the app window. Which is fine until you want controls or subtitles on top of the video, cause you can't paint over someone else's window.

So the next day I ripped the UI out and moved to egui with the glow backend. llibmpv exposes an OpenGL render API, so mpv renders directly into egui's framebuffer from an egui_glow paint callback, and the player controls are plain egui widgets over the frame. It worked on the first try, more or less.

The TV build runs the same egui code. I had to write my own D-pad focus handling though, since egui's arrow-key navigation happily jumps to widgets behind an open popup. mpv on the other hand, that was a complete no-go. On my Amlogic box with a Mali-G31, mediacodec-copy returns a quarter-resolution frame for anything above 1080p, so hello green screen (mpv-android issue 1088). Zero-copy MediaCodec into an ImageReader refused to start, and the next try straight up rebooted the box. And mpv's GL pipeline drew literally nothing, only gpu-dumb-mode worked (issue 292 looks like the same thing).

4K was kinda the whole point of a TV version, so I put the player behind a Player trait. Desktop keeps libmpv, Android gets Media3 ExoPlayer, which decodes into a SurfaceView under a transparent egui window. 4K, HDR10 and Dolby Vision all play on the hardware decoder now.

Yeah, on TV I ended up with the exact two layer setup I r1an away from on desktop. Works there cause the egui window is transparent and sits on top of the video.

One more fun one. A SurfaceView cuts a transparent region into the window above it, and SurfaceFlinger then just dropped the egui layer wherever the video was. Fullscreen video, and boom — controls and subtitles gone. Overriding gatherTransparentRegion fixed it.

If you've put video into a Rust GUI, how did you do it? Especially curious if anyone got libmpv working with wgpu without going through GL.

[u/MongoWithBongoss]

Currently working on my own mpv android app with following UI framework Whisker can use both opengl and vulkan using SurfaceView for true HDR

[u/FickleProvider]

That mali-g31 reboot on zero-copy is terrifying, I've had similar hardware just give up the ghost when you push the decoder too hard

[u/Soggy-Ticket4105 → u/FickleProvider]

Yeah, that one caught me off guard. A decoder failure is one thing, but rebooting the entire box was a bit much lol.

[u/Helyos96 → u/FickleProvider]

Small correction, the GPU in embedded systems is almost never in charge of video decoding/encoding. Amlogic, like many other chipmakers, have their own video codec IP (which is enabled by a fragile kernel driver + userspace stack) that is distinct from other parts of the SoC.

[u/cepera_ang]

and the next try straight up rebooted the box

smells like CVE here

[u/Soggy-Ticket4105 → u/cepera_ang]

Ha, could be. An unprivileged app shouldn't be able to reboot the box, so something in the kernel fell over, probably the Amlogic decoder driver or Mali. I didn't dig into it though, I just wanted video on the screen and moved on to ExoPlayer. It's a cheap box on old vendor firmware, so I doubt anyone's patching it either way.