ffmpeg-video-composer v3: Cards in HTML, Sounds in Code, Renders on the Phone

In this piece

LeClap turns a JSON template into a finished video, and renders it wherever the footage already is: on a computer, in the browser, or on the phone. Its engine, ffmpeg-video-composer, is now at v3.

The short version: you can lay out a card in HTML instead of positioning text by hand, compose a sound effect instead of hunting for one, caption speech without uploading it, render one template in three formats, and have the builder refuse a template before it wastes a render. Release 3.2.0 adds masks, blend modes, motion blur and filled vector shapes, lets the elements of an HTML card move one after another, and the template builder has been redesigned around the music and the timeline. The film below runs just under a minute and a half, and every result in it is a real render.

Cards like a web page#

Until now, a lower third in LeClap was a stack of drawtext and drawbox filters, each placed by coordinates. That works the way a spreadsheet works as a drawing tool.

An input of type: "html" now takes html, css, a width and a height:

{
  "name": "address_card",
  "type": "html",
  "html": "<div class=\"card\"><span class=\"tag\">For sale</span><h2>{{ address }}</h2></div>",
  "css": ".card { display: flex; flex-direction: column; padding: 26px 32px; border-radius: 28px; background: $color.surface@0.92 }",
  "width": 560,
  "height": 300
}

The CSS is a flexbox subset, not a browser: no grid, no CSS animations, no calc(). Motion lives next to it instead: tag an element data-motion="tag" and the input’s motion map moves it (x, y, scale, rotate, opacity, stagger), so a card’s tag, lines and chips come in one after another. The card is drawn at the output frame rate over the animated span only, bounded to 8 seconds. The markup is sanitised (no scripts, iframes, forms, SVG, event handlers or links), and {{ address }} is filled HTML-escaped, so copy can never smuggle markup in. Satori, resvg and HarfBuzz draw the layer at 2× into a transparent PNG named by its content hash, and from there it is just an image that moves like any other.

The useful part is that it is one pipeline on every host: the same code draws the layer on Node, in the browser and on the phone. Only the WebAssembly’s address changes. On the phone that address is unusual: Hermes, React Native’s JavaScript engine, has no WebAssembly, so the build ships a self-contained 5.2 MB page that a hidden WebView runs to draw the layers. Nobody said “native” had to mean “no browser anywhere”.

A follow-up release lets templates bring their own fonts for these layers (global.fonts: TTF, OTF or WOFF). WOFF2 is refused, with the command that converts it, because neither Satori nor HarfBuzz reads it.

Sound effects, made to measure#

A sound effect cue can now take a sound instead of a file: layers of tones, noise and strikes, shaped by envelopes and filters, at most 4 seconds and 8 layers. A small synthesiser written in TypeScript renders it to a WAV, seeded, so the same sound and seed give the same file.

On a given platform, that is. Node, browsers and Hermes each compute Math.sin their own way, so the last bit can differ between them. The changelog says so, which is more than most audio tools admit about anything.

Captions, written for you#

subtitles.transcribe asks for a section’s speech as word timings. On Node, whisper.cpp transcribes it locally, and the result is pinned into the template: the words, the model, the language and a digest of the clip. Renders only ever read pinned words, which means a caption doesn’t change because the model woke up in a different mood.

The browser and the phone don’t transcribe; they report transcribe_unavailable and render pinned words like everyone else. The model is downloaded once, on explicit opt-in, checked against its SHA-256, and never bundled. If you edit the clip after pinning, transcript_stale tells you the captions now describe a video that no longer exists.

One template, three formats#

A vertical cut of a wide film is a different composition, not a crop. A template can now declare formats: patches applied when it renders portrait or square, plus { "$format": { … } } values for the numbers that change per aspect. Validation runs once per declared format, and format_crop_only points out a format that changes nothing, in case you were hoping nobody would notice.

Typed fields, checked before the render#

global.fields declares what a template expects (text, colour, URL, media, number, enum, time), with defaults, limits and required. A value that is missing or of the wrong type fails the render before encoding, not halfway through it.

That matters most in the web builder, where the built-in AI writes the template from a one-sentence description. In the film, the generation takes 1.2 seconds. On screen it says “sped up”: the real one took four and a half minutes. Either way, the fields it writes are checked like anyone else’s.

Also in this release#

• Masks and track mattes. An image, animation, html or video layer takes options.mask: a shape (rect, ellipse or an SVG path) or another layer read as a matte, by alpha or luma, with invert, feather and a mask that moves with the layer.
• Blend modes and motion blur. options.blend (multiply, screen, overlay, add, difference and friends) mixes a layer into the picture inside its own alpha. motionBlur supersamples only the frames that actually move, so resting frames stay sharp.
• A shape graphic. A filled vector shape (star, polygon, heart, arrow, seeded blob, rounded rect, or your own SVG path) with a colour or gradient, an outline that can draw itself before the fill fades in, and the same keyframe tracks as any layer.
• The app templates were restyled. Five of them (Interview, Product Launch, Web App Promo, App Tutorial, Story Reel) now draw their cards with HTML layers. The film plays Interview and Story Reel before and after.
• Effects. Thirteen procedural light and texture effects (sheen, bloom, confetti, grain…), each with an on-device fallback, and 14 kinetic typography presets laid out with the real metrics of the bundled fonts.
• An effects tour. One template, six minutes, every motion effect in ten chapters.
• Also in 3.2.0. Clip boxes, bands, shaped iris and zoom-through transitions; reading the format of a reference clip; brand kits fetched from a website; and subtitles rendered as HTML.
• FFmpeg 9. It removed -filter_complex_script, which the engine relied on for long filter graphs. It doesn’t anymore.

And the reason this is a major version rather than a minor one: validation is stricter (unknown keys are now errors instead of being silently dropped), and renders are no longer byte-identical to 2.5.0, because a deterministic encoder profile is now on by default. If you pinned golden files, pin them again. The migration guide has the details.

A builder that shows the music#

The web builder has been redesigned around what a video is made of. Music is a track list with a waveform, BPM and mood for each track, and a Default track for templates that don’t choose. The transport is frame-accurate. The scene strip is sized by duration and has an audio lane, so you see where the music sits against the scenes. Fields have types and defaults, and a preview shows the form a viewer will actually fill in. The studio around it got the usual round of polish: the sort of changes nobody notices until they are missing.

On the phone, no server#

The same engine runs in the LeClap app, on the device (transcription aside, as above). In the film, the render is a screen recording of the real app, a Release build, on an iPhone 16 Pro simulator, labelled as such. It picks a template, renders it and plays the result: 72 seconds, on a heavily loaded Mac pretending to be a phone. There is no upload step to show because there is no upload.

Your own assistant, handed over as a link#

An outside AI assistant can write a template too, then pass it on as a link to the builder. The template travels in the URL fragment, which browsers never send to a server. To quote the docs: “Nothing is uploaded, and LeClap has no server to upload to.” Media that only exists on the author’s machine can’t travel in a link, so the builder lists what needs to be picked again.

The same assistant can also prove its own work. When it opens a pull request that changes a screen, the repo’s evidence template turns two recordings, the base branch and the branch, into a before-and-after video, side by side and frozen on the change, rendered by LeClap and attached to the description. The template and the script that builds it live in examples/agentic-pr-video/evidence-skill. Reviewers get to watch the change instead of trusting the summary, which is a novel experience for everyone involved.

Upgrading#

The engine is on npm as ffmpeg-video-composer, with the leclap CLI alongside it. Templates that don’t use the new features render as before, apart from the stricter validation and the byte-level differences above.

---

All of it is MIT: github.com/heristop/leclap. The builder, the samples and the effects tour are on leclap.dev.

v3 adds a lot of surface: HTML, a synthesiser, a transcriber, formats, typed fields, masks, shapes and a builder that finally shows the music. The part I care about is that each one ends the same way as before: a plain FFmpeg command, run on whatever device holds the footage.

---