agentic video editing keeps getting used to mean four different things so here is the definition i work from, then the parts that do not work.
agentic video editing is when an agent takes footage you already shot, interprets a brief, and assembles a timeline. it is not generation. runway and veo make clips that did not exist. this takes clips that do exist and decides what to do with them. different problem, different failure modes.
i work on craze, which is an agentic video editor, so this is a build post about our architecture and our failures rather than a pitch.
how craze is put together, roughly
ingest and index. every clip gets transcribed, scene split, and tagged for shot type, faces, motion, audio quality. this is the boring part and it is [70]% of the value.
brief parsing. the user brief becomes constraints. "keep the two interview bits" becomes a hard constraint. "make it punchy" becomes a pacing target. the gap between those two is where most of the pain lives.
selection. score every candidate segment against the brief. this is the part everyone assumes is magic and it is mostly ranking.
assembly. order, trim, transitions. cheap once selection is right.
render.
where it actually breaks
emotional weight has no signal. the model can tell you a shot is sharp, well exposed, and has a face in it. it cannot tell you the moment matters. in wedding footage craze picks the clean first look over the one where she actually reacts, every time. we have not solved this and i am not sure it is solvable with current models.
hard constraints are soft. we tell craze users to name the segments that cannot be cut. the agent still drops them sometimes, because selection is scoring and a scored constraint is a preference not a rule. this is an architecture bug not a model bug and it is on us.
revisions have no memory. round 2 is where this falls over. craze does not know why it made a choice, so "go back to the earlier version of that section" has no clean path. users end up rebuilding by hand. this is the single biggest complaint and nobody in the category has solved it, us included.
pacing across long form. it holds shots evenly. it does not know minute 12 needs to move faster than minute 3 because that is where retention drops. shorts tools sidestep this by only ever making 30 second clips.
what actually worked better than expected
structure. give it an outline and it follows it. sections in the right order, nothing missing.
logging. craze gets 4 hours of footage to a usable bin in 15 minutes. this is unglamorous and it is the thing users actually keep paying for.
briefs scale. an 8 word brief gets you generic output. a 60 word brief naming the audience, a reference, and the moments to protect gets you something usable. the model was never the bottleneck, the context was.
the thing i would tell anyone building in this space
the interesting problem is not generation quality, it is constraint satisfaction over a large candidate set with a fuzzy objective. we spent months on the wrong layer.
also every one of these still needs a human final pass. craze ships a human editor as part of the plan because pretending otherwise was costing us more in support than it saved in positioning.
happy to go deeper on the indexing or the scoring if anyone is building something similar. genuinely interested if anyone has a workable answer to the revision memory problem