Skip to content
Christian Andrey Reyes

ShortsWithAI

Turns long videos into captioned shorts and publishes them to four platforms.

ShortsWithAI

Pitch video in production

Role
Solo developer
Period
Jun – Nov 2026
Stack
Next.js 16, TypeScript, ffmpeg, whisper.cpp, Transformers.js, SQLite, Drizzle, React 19, yt-dlp, Cloudflare R2, Vitest, MSW

The problem

Cutting a long video into shorts by hand is slow: find the moments worth keeping, reframe a landscape picture to 9:16 without losing the speaker, caption it, then post the same clip to four platforms with different APIs and rules. I wanted one tool that handles each step and still leaves the choices to a person.

What I built

  • A background job queue on SQLite with retries and backoff, so ingest, transcription, rendering and publishing never depend on an open browser tab.
  • Moment finding that merges YouTube's most-replayed heatmap with LLM-scored transcript segments, plus an audio-energy detector for uploaded files.
  • A speaker-following crop: ffmpeg shot-change detection splits each clip into shots, frame motion finds the speaker, and a vision model settles unclear shots.
  • Rendering of 1080x1920 clips with burned-in captions, and a timeline editor for order, trims, transitions and hand-edited subtitles.
  • Publishers for YouTube, Facebook, Instagram and TikTok, each an independent job with its own retries, status and post link.
  • OAuth connections to each platform, with stored tokens encrypted at rest using AES-256-GCM.
  • An LLM provider chain (Gemini, OpenRouter, Groq and others) that falls back to a local LM Studio model.

How it works

  1. 01

    Ingest

    A YouTube URL through yt-dlp, or an uploaded file; whisper.cpp transcribes locally.

  2. 02

    Find moments

    Replay heatmap peaks, LLM-scored transcript segments and audio energy, ranked in one grid.

  3. 03

    Cut and reframe

    ffmpeg cuts each moment to 9:16 and the crop follows the speaker shot by shot.

  4. 04

    Caption

    Subtitle cards from the transcript are styled and burned into the frame.

  5. 05

    Edit

    A timeline for order, trims, transitions, b-roll stills and a watermark.

  6. 06

    Publish

    One job per platform, polled until the platform confirms or rejects the post.

Decisions

Motion first, vision model second

To find the speaker I measure how much each column of the frame changes between samples, which needs no model. Only the shots it cannot call go to a vision model. Sending every shot to the model was the alternative; it costs a request per shot, and most shots do not need one.

One job per platform

Publishing to four platforms runs as four independent jobs, each with its own retries and status. A single job that posts everywhere was simpler, but one rejected upload would have blocked or repeated the other three.

Leave a b-roll cue empty rather than guess

When no library image matches a cue closely enough by CLIP similarity, the cue stays visible and unfilled. Filling it with the closest image always produces something, but often the wrong thing, and it hides which image is missing.

Evidence

This repository is private, so these figures can't be checked from outside. I can walk through the code on request.

  • 25 commits, all mine
  • 89 tracked files under tests/ (Vitest, with MSW for platform APIs)
  • Seven features, each with a written design spec
  • Publishers for YouTube, Facebook, Instagram and TikTok

Next

  • Build a labelled set of clips and measure how often the motion pass and the vision model pick the right speaker, then tune the thresholds against it.
  • Move rendering to a separate worker process so long renders do not share a process with the web app.