“Automate video editing” covers three different jobs, and most of the disappointment with it comes from mixing them up. Being specific about which one you mean is most of the work.
Three jobs, one phrase
Scripted transforms. ffmpeg, moviepy, a shell loop. Resize a folder, concatenate, normalise loudness, burn in a caption file. Deterministic, no judgement involved, and it has worked for twenty years. If this is all you need, you do not need a model.
A harness that plans and executes. Something reads your footage, proposes an edit, and carries it out after you agree. This is the part that is new, and the part this article is about.
Driving the editor you already own. Premiere, Resolve, Final Cut, through whatever API that application exposes. This one is gated by the vendor rather than by your tooling: what you can automate is exactly what their scripting surface allows, and that surface moves between releases and between free and paid tiers.
They fail for different reasons, and a tool that is good at the first is not automatically good at the second.
The line editors themselves draw
It is worth reading what editors say when this comes up, because they are more precise about it than the marketing is. The most-upvoted answer in a recent r/VideoEditing thread on AI editing put it plainly: the value you provide is knowing what to edit, not controlling the software. In a parallel DaVinci Resolve thread, the person complaining loudest about AI being pushed into editing tools drew his own line without being asked — transcription, rotoscoping, object removal, audio cleanup, tracking, boring repetitive tasks: fine, if it saves time.
That is a usable specification. The mechanical layer automates. The decisions — what to cut, when to cut it, which take carries the moment — do not, and a tool that pretends otherwise produces work you have to redo.
Evidence before cuts
The common failure is prompting straight to an outcome: “make a highlight reel from this.” The model has not seen the footage; it is inferring structure from your sentence.
The step that fixes it is unglamorous. Before any cut, extract what is actually in the file:
- Transcript with timings — and speaker labels when more than one person talks.
- Scene boundaries — where the shot actually changes.
- Silence spans — where nobody is talking, at a threshold you set.
- Quality flags — clipped audio, badly exposed stretches, frozen frames.
Now a cut list is a decision over data rather than a guess, and — this matters more — it is reviewable. You can read why a segment was chosen before you spend render time on it.
Plan, approve, assemble
The second thing separating usable automation from a party trick is that the plan is a separate artifact from the execution.
Concretely: the run produces an edit decision list first — every segment, its source file, its in and out points, and what happens over it. You read that list. Only after you approve it does anything render. Three things follow from the split. A mistake costs you a read rather than a render. The list is a diff, so a second pass changes four segments instead of redoing everything. And when the output is wrong, you can tell whether the plan was wrong or the execution was — different bugs with different fixes.
If a tool renders straight from a prompt with no reviewable intermediate, it has automated guessing rather than editing.
What does not automate
- Story and taste. Two editors with the same footage tell two different stories. That difference is the job.
- Anything that needs to feel the room. Which take is funnier, which pause earns its length.
- Audio-only or transcript-only deliverables. Worth stating plainly because it is a common misfit: a video pipeline is not a transcription service, and using one as the other gets you a worse result than the dedicated tool would.
The cost question, honestly
Editors raise this before they raise quality, and they are right to. Anything model-driven carries a per-run cost, and long footage is expensive to look at. Extracting evidence once and reusing it across passes is not only tidier — it is most of the difference between a workflow you keep and one you abandon after the first bill. Prefer a deterministic step wherever a deterministic step will do.
Running it in Orkas
Orkas ships VideoStudio, which works in exactly this order: evidence from the footage, then a cross-modal edit decision list, then an approval gate, then deterministic assembly — picture, overlays, narration, captions, loudness checks. It is open source and runs on your machine, and the same desktop drives the coding CLIs you already have installed, so an editing run and a scripting run are the same kind of task rather than two separate tools. The scenario walkthrough is the video production workflow use case.
What it deliberately does not do: deliver audio-only or transcription-only output, or render anything before you have approved the plan.