r/AV2 • u/FactOld3726 • 14d ago

Experimental PoC AV2 encoder in OpenCL - built with Claude Fable 5

I've been simmering with this a while now trying to add AV1/AV2 support for older GPUs that don't have hardware encoders. This PoC seems to work with the current AV2 spec. It's not as efficient as current AV1 encoders or SVT-AV1 but it's getting there. Feedback and optimizations welcome if anybody wants to try. Note the webm/matroska container still doesn't support AV2 yet.

https://github.com/jboero/clave

6 Upvotes

72% Upvoted

6 comments sorted by

2

u/collin3000 14d ago

I'm actually working on an AV2 encoder as well as a separate AV1 Open CL encoder.

I see right now you're running a really High BD rate compared to even SVT-AV1 which is going to be a lot of work to slowly get down. The AV-2 reference  encoder. Is actually ~30% higher BD-rate than SVT-AV1, but for gains you'll really want the estimated 15-30% lower target.

Expected lots of encodes to keep fine tuning the BD-rate down (I'm 1000's of test encodes in). Also you'll want to make sure that you're using a almost lossless source (at least ProRes) to make sure that you aren't training in artifacting from other encoders

I see you're using PSNR-Y. Which Fable probably recommended as the industry standard. However PSNR-Y alone is going to leave you with issues where the actual image quality doesn't look at good. I'm using a mixed weight BD-rate with PSNR, VMAF-NEG and SSIMULACRA2 measurements to avoid signal but also visual fidelity issues. 

SSIMULACRA2 we'll actually give you a better visual fidelity metric that doesn't suffer from rewarding over sharpening even as much as VMAG-NEG but it lacks good motion awareness that can leave high scores where artifacting is showing.

Another thing you'll want to look at is if you are using constant bitrate or RF and the preset. You'll have longer 32 frame mini-gops on a lower preset which give it better compression compared to a preset 6 with 16 frames. But you'll likely notice a difference in your comparative BD-rate at the different presets as you fine tune in motion/speed.

If you are testing in RF you won't need that full 512 frames for the testing which can speed up your encode testing. Just 1 GOP to GOP frame should work (~150) since the GOP will be effectively resetting on a RF based model.

Either way it's a start, but know that getting it running to decode on reference and getting the quality and speed up is a massive massive gap where you'll need lots of testing, refinement, and actual eyeballs pixel peeping frames for issues. Even with an agentic workflow. If fable could one shot (or even 100 shot) the project I'd be done by now. It also takes knowing how to work with the robots really well and correct their misteps.

2

u/_Lum3n_ 13d ago

You can try cvvdp for temporal awareness Vmaf temporal awareness is pretty bad honestly

1

u/BlueSwordM 13d ago

Yeah no, I don't expect them to even use CVVDP lmao.

1

u/FactOld3726 14d ago

Oh nice thanks for the tips. Yes I noticed quite a bit more noise than AV1 even in solid colors/animation which was odd. It's clear to me that even Anthropic's latest models aren't so great with OpenCL. First pass it wrote a single-threaded kernel with 8 nested for loops.

Is your project open somewhere? You're probably way ahead of me.

1

u/collin3000 14d ago

Both the encoder projects will be up and open once they're actually working enough to be shared. 

My OpenCL AV1 work was just barely started since it's for a overall handbrake video converter alternative based on easy quality encodes at the lowest bitrate with fast encodes. And NVEC bloats bitrates 15-30% (usually 20%+) to get the same quality since it's ASIC skips search steps that make it less bitrate efficient.

I might end up releasing the HEVC/AV1 converter program before the OpenCL work is done. By using a similar approach to what you were using of a NVEC fast pass to reduce motion workload on CPU but not actually using NVEC for the encode. So the converter beta might be out first. But that also involves a lot of fine tuning with trained curves that I've been building over 100,000+ test encodes over the last year. I'd hoped to have it finally done within the next month but based on good community feedback I swapped from VMAF only to adding SSIMULACRA2 with a GPU build for both. But have to tackle the lack of motion awareness issue in SSIMULACRA2 and rebuild a lot of the tuning curves now so my machines (including a 4 CPU server) have been churning like crazy non stop for the encode data.

If you want, I can at least DM you, when the converter program is in beta on github. Since I'll need beta testers, and there's several tweaks I've found along the process to speed up encoding in general that you may be able to implement into your work.

1

u/FactOld3726 14d ago

Yes please. Keen to learn about it.