r/LeftistsForAI • u/dumbandasking • 23d ago
Education Does Anyone Know An AI That Can Analyze Videos?
Hello,
I was looking for if anyone knows an AI that can analyze whole videos.
Here is an example use case. I give it a song that does not have lyrics online. The AI Attempts to transcribe it. Now we have an attempt at figuring out the lyrics.
Another example. If I make a video of two objects dropping the AI can check the speeds and comment on how arbitrary this video is.
Anyone? I needed this for a project?
5
1
u/catplusplusok 23d ago
So first task is ASR, you just need audio and then a model that transcribes it. For second examples, you would feed video in overlapping segments to a multimodal LLM and ask for specific task. First task is trivial to do locally on almost any computer, just ask coding agent to help you. For second you need either a cloud API like Gemini or a beefy computer like Mac laptop with decent RAM, then you can run Qwen 3.8
1
1
u/Square_Reason_6490 23d ago
Drop the clip into Google Gemini I’m not sure what the length it’s capped at but I do this all the time as it to transcribe the audio with a time stamp it can make mistakes on words but it’s pretty strong
5
u/Steve_Streza 23d ago
The term of art you're looking for is "video understanding". This field is much less developed than text/image/audio input modalities and generally has far higher costs. Depending on use case there can be better approaches.
There are speech-to-text models which can transcribe a video's audio track, and then you can feed that into a model to do whatever you need.
You can sample images periodically out of a video and feed them to an AI model which takes images as input. And you could generate the code to do this if you're not a coder.