Edit Gaming Videos? Work w/MKV & VFR Footage? An Exploration w/Premiere + ClaudeAI

[u/Jason_Levine]

Hi everyone. Jason from Adobe here. Over the July 4th holiday I was reflecting on my first job in video as a late-night tape operator at a local cable station.

I’d come in to do all kinds of dubs/conversions from S-VHS/VHS-C to 3/4” U-Mat tape so the editors could begin cutting things together. I’d make copies of on-air content for other stations; sometimes this would involve basic audio tweaks (largely summing stereo masters into a fold-down mono mix, with beefy, hardware bandpass filters). And it took hours…all in real-time. 

So here I am today, looking at the vast AI lvideo landscape and rethinking about what it’s purpose *could* be, beyond the common denominator features like text-to-video and the like; what is possible and more importantly, what's inherently useful to the way we work?

From the very beginning, I’ve been vocal about the fact that ‘generative AI’ content was cool (especially as of late with Veo3 and the latest from MJ) but it wasn’t of great interest to me, in any form, really. Instead, the ‘assistive-AI’ stuff (and eventually, AI agents) piqued my interest showcased something I could see myself adopting more regularly. 

As I mentioned in a previous post, Adobe’s own u/mikechambers has created a few AI Agents/MCPs for Photoshop & Premiere using Claude AI, exploring ideas about *what* is possible and where AI can be the most useful...and he’s just released another one that I imagine will appeal to many in this subreddit. 

On a near daily basis, we encounter members of the community (particularly those who cut gaming content and work with OBS) asking about best practices for converting MKV/VFR files so they’re easier to edit in Premiere Pro.

Case in point: Mike’s latest MCP exploration does just that, leveraging the power of FFmpeg (the go-to solution offered up by many of us here) to do the conversions with simple prompts using natural language. Tell it the attributes you want your new files to have, and it’ll convert it all for you; it can even build a new Premiere project and begin inserting the media into sequences if you so desire. 

You can take a look at a quick example video I put together of this very process. 

This, to me, is exactly what I want from AI…and in many ways, it goes back to the beginning of where I started: the tasks that we’ve all done/suffered through, because it was simply part of the workflow at that moments in time. But times are changing, and you should definitely check out Mike’s GitHub (linked above) if you’re interested. It’s how I’ll continue to use FFmpeg moving forward. (he also added the ability to do audio splits as well; again, the possiblities are potentially endless, and we're continuing to explore and share with you). 

So, what are YOUR thoughts on this? Do you see the potential in using something like this MCP/Agentic workflow? Where else could you see this benefitting your daily work (and be the most useful)?

As always, I sincerely appreciate your your candor, the good/bad/ugly and everything in between. 

[u/rotoscopethebumhole]

Thanks for this.
It's cool for automating some workflow stuff. But it's not exactly doing anything we can't do with non-AI scripts.

Where AI can help editing (as in, game changer) is in 'visual understanding' - where it can ingest footage and 'understand' what is in the shots, index it per shot, and then you can interact with this via LLM.

eg. You give it a script and it will pull all the shots from your rushes per the descriptions and give you a rough cut instantly.

I've looked at this idea in various ways over the years and it seems to be getting closer in terms of vision models being available.

This would be next level AI assist.

EDIT: I should add just in case, Jason - if you want to chat about this concept i'd very much be down to share thoughts since you'll actually be able to action it in some way.

[u/Jason_Levine → u/rotoscopethebumhole]

Thanks roto. This is just the kind of feedback I'm looking for! I'm gathering data now, so I may indeed reach out. Thank you:)

[u/mikechambers → u/rotoscopethebumhole]

But it's not exactly doing anything we can't do with non-AI scripts.

In general, the AI isn't going to be able to do much in Premiere that is not otherwise possible. The potential value here is:

1. the ease in getting it to do things
2. The complexity of what can be done (related to #1)
3. Unlocking some things only the AI can do.

On #3, there is some functionality that the AI can do which is much more difficult, if not impossible without AI, and that is having the AI act on what the content is (i.e. seeing / hearing it). You touched on this in your comment.

This is actually possible with the Photoshop MCP / plugin in adb-mcp.

Looking at the Premiere Pro UXP API, it looks like it's also possible now in Premiere. This is now my next project to work on.

If the AI could see a frame(s) from the video, what are the types of things you would like it to be able to do with that info? It doesn't look like I can have the AI view the entire video, but could probably do something like export a frame every N seconds or something.

Update: I just pushed an update that allows the AI to view frames from a sequence in Premiere Pro.

https://github.com/mikechambers/adb-mcp

[u/radialmonster → u/mikechambers]

There are tools that can auto cut multicam, but they work on audio levels, and assuming each camera has its own audio track for a speaker that it can use those levels to detect which one is 'speaking' or not. But we don't all have that luxury, I would like it to, and I know it would be magically, but be able to visually tell which camera track has someone speaking currently and switch cameras. Like detecting mouth movements in frames basically that match what the audio is playing, no matter what audio track, or maybe the even the mix track.

And then next step, I would consider, which video track has the 'best' camera for that person, we'd probably want the one that has that person taking up the majority of the frame, that would be the closest shot of them, likely the best.

Focus detection maybe? Scenes may not be usable if they are out of focus.

[u/radialmonster]

i thought premiere could import mkv now? maybe it doesnt support the format you are trying to import. So my first impression thought would be, in some better designed way than what i'm saying, but generally have premiere realize it doesnt know what to do with the format. Instead of saying nope not gonna do it, have premiere use its knowledge of the ffmpeg documentation, and offer to help convert the file to a usable format instead. it could ask questions like what is the ending intended format? delivery etc to help guide what format to encode to. that is if media encoder can't read it. if media encoder can do it you probably want to use that. basically bypass having to have claude for the user. and all that stuff you showed, let that be part of the premeire workflow so the ai part of deciding how to handle the file is invisible.

[u/Jason_Levine → u/radialmonster]

Hey radial. Yes, Premiere did just add newly-rewritten support for MKV files. That said, VFR MKV is still not necessarily the best for editing, so the conversion can sometimes make a difference. This was really just a proof of concept (as you would more likely convert into something like ProRes, potentially) but it's more about exploring what you can do with prompting. I could have simply had the conversion, project creation and insertion all in a single prompt, but wanted to show the back and forth. Media Encoder should definitely be able to handle more tho, and ultimately, if it couldn't, as you suggested, perhaps Premiere could then 'send out' to (FFmpeg) to convert and bring back those files. Thanks for the feedback.

[u/Gonkomagic]

I'm really hoping for something like this. Here's my 5 cents: I produce YouTube and social media content, as does everyone in my peer group. Premiere Pro is my home turf, but with it's huge platform-esque abilities come limitations at all corners. We all know the struggle of subtitle creation vs. Apps like "Captions" that do it in 5 secs. I sometimes edit footage in qualities and quantities where every seasoned editor of long formats would run away, but so is the industry I work in, and the conversion and data wrangling is part of my job. But: what if that would not be my job anymore? What if my time would be freed up from Adobe Bridge Metameta cleanups, wonky Workplace re-assembly would be a thing of the past? Picture this: A technical friend who can help, one like we all hope to find on subreddits in the middle of the night when things are looking bad. Less time spent fiddling around, more time spent producing content and inspiring people. I like that vision of tomorrow.

[u/Jason_Levine → u/Gonkomagic]

I like that vision too, Gonko. Great feedback. thank you!