Give the Agent the Whole Video as One Image

Frank and I meet every now and then. Sometimes on a Friday, sometimes on a Monday. We drink a lot, talk for hours about art and things that are almost art, and then we drink more. When it’s time for the next round, I send him a message, usually with a gif.

This time I sent him a website instead. You scroll, a tall figure sways like a pendulum, a hole opens in his chest, and at the very end his trousers drop and a cartoon penis swings in time with the balance. That is the whole joke. It is not what this post is about.

This post is about how it got built, because the trick underneath is reusable:

1. A video is a hard target for an agent. One image of the whole video is an easy one. Tile every few frames into a contact sheet and the agent sees the entire choreography at once.
2. Measure the motion, don’t describe it. A short numpy script over every frame turned “he sort of sways” into curves the animation reads directly.
3. Then it’s a conversation. One big prompt got a working scene. About ten short follow-ups made it right.

Frank first, though.

Frank, and why this is a homage

Frank Nitty 3000 is an artist and director. His bio says “artist • director • NotAI”, and he means it: his work is made by hand, with his taste, by a person who has been doing it for years. Around 38K people follow him on Instagram. He’s also the founder of Studio 3000 , a digital production house he started in 2018: video animation, show projections, store window screens and digital out-of-home for the lifestyle, fashion and luxury industry, with a bilingual English/Chinese team that also makes content for Douyin, Weibo and Little Red Book.

Let me be plain about it. The idea in the original reel is Frank’s. The composition, the strangeness, the decision to hang a sneaker above a head and a ring of faces around the shoulders: that is taste, and you can’t prompt it out of thin air. What I made is a goofy beer invitation built from his piece, with an agent doing the carpentry. It is not “AI can do Frank”. It can’t. I know because I had to supply the target.

The reel is here . His caption says Dries Van Noten is his favourite designer, tagged #surrealism. Visually: a full-height figure on a pale-blue studio seamless, a hole in the chest that opens into a tunnel with a floating pink sphere, a sneaker hanging on strings above the head, a ring of face fragments hanging around the shoulders. The figure sways like a pendulum for 23 seconds.

Go watch it first. Then scroll my version: **egor3000.egor.lol** (works on a phone).

Six stages of my version. Yes, one frame has the trousers down. Frank will survive.

The pipeline in one picture

flowchart LR
  A[Reel on Instagram] --> B[Headless browser grabs 1080p stream URL]
  B --> C[ffmpeg: 695 frames]
  C --> D[Contact sheet: 47 frames in one image]
  C --> E[numpy: measure motion over all 695]
  D --> F[Agent prompt: layers, timing]
  E --> G[JSON curves]
  G --> F
  F --> H[three.js scene]
  H --> I[Static files on a Cloudflare Worker]

Step 1: getting the video (the part that fought back)

I started the boring way. yt-dlp hit Instagram’s login wall, even with my browser cookies handed over. So the agent skipped the downloader entirely: it opened the reel in a headless browser, watched the network log, and pulled the 1080p video stream URL out of it. Then a plain download.

If a downloader fails, don’t fight the downloader. Ask the browser what it actually fetched.

Step 2: every frame, then one picture

The reel is 23 seconds at 30 fps. ffmpeg turned it into 695 frames, and then into one image the agent could read. These are the commands that were run:

# every frame
ffmpeg -i reel.mp4 frames/f_%04d.png          # 23 s x 30 fps = 695 frames

# one image the agent can read: every 15th frame (2 fps), 8x6 grid
ffmpeg -i reel.mp4 \
  -vf "select='not(mod(n\,15))',scale=270:-1,tile=8x6:padding=4" \
  -frames:v 1 sheet.jpg

Every 15th frame is 2 fps, which gave 47 frames in an 8×6 grid. That one image is what the agent “looked at”.

Frames from _Frank Nitty 3000’s reel_ , © Frank Nitty 3000. All the frames in one picture: what the agent saw.

Photographers have done this for a century. A contact sheet lets you read a whole roll in ten seconds: what repeats, what changes, where the good frame lives. An agent reads it the same way. From one image it worked out the layers (figure, chest tunnel, sphere, hanging objects), what moves and what never does, and roughly when things happen.

Compare the alternative. “Here’s a video file” gives an agent nothing it can grip. “Here are 695 images” blows the context. One sheet is the sweet spot: dense, ordered, cheap.

Step 3: measure it, don’t eyeball it

A contact sheet tells you that the figure sways. It doesn’t tell you how much, how fast, or where it bends. If you let the agent guess, you get a generic sine wave, and the balance feels like a screensaver.

So I had it measure all 695 frames. The method is simple: find the background colour, mark everything that isn’t background, and track where that shape sits at a few heights (feet, knee, head).

# measure the motion instead of eyeballing it
import numpy as np, glob
from PIL import Image
for f in sorted(glob.glob('frames/f_*.png')):
    a = np.asarray(Image.open(f).convert('RGB').resize((270, 480)), dtype=np.float32)
    bg = np.median(a[:, :20].reshape(-1, 3), 0)          # studio background colour
    m = np.abs(a - bg).sum(2) > 60                       # everything that isn't background
    cx = lambda y0, y1: np.nonzero(m[y0:y1])[1].mean()   # horizontal centre of a band of rows
    feet, knee, head = cx(401, 426), cx(306, 326), cx(120, 150)
    lean = (head - feet) / 326                            # head offset in body heights

Save feet, knee, head and lean per frame to JSON and the animation can read them directly.

What came out:

That table is the spec. I’d never have written “fast snap right that overshoots and settles” from watching, and it’s exactly why the original feels alive. The curves went into JSON and drive the animation, so the balance behaves like Frank’s, not like a generic wobble.

Step 4: the build

The scene is three.js, in layers:

• The figure is a flat plane, bent from the feet in a vertex shader. Feet pinned, bend weighted low, as measured.
• The chest hole is a real 3D tunnel with the sphere, a ring and a moon inside. That gives true parallax when the camera moves, which a flat cut-out would never give.
• The banana. I swapped the sneaker for a procedural 3D banana. My joke, not Frank’s, so it should not pretend to be his object.
• Eight different hand-drawn faces on strings, and the strings touch the faces exactly.
• Arms and the pendulum run on damped springs. Trinkets and dust float around.

Scroll drives a timeline with eight beats: Stand, Open, Crown, Orbit, Turn, Bloom, Drop, Dive.

The loop that never ends

At the end the camera dives into the chest planet until its colour fills the screen. That colour becomes the next world’s sky, and a new guy rises in from below. Every planet is seeded-random: outfit, faces, trinkets, banana size, even pendulum length (sometimes tiny, sometimes dragging on the floor). On mobile the page is locked and only the scene moves, with a virtual scroll and touch momentum.

It deploys as static files on a Cloudflare Worker. No backend, nothing to babysit.

What “almost one shot” actually means

The first prompt plus the contact sheet produced the working layered scene. That part still surprises me.

Then came about ten short follow-ups. “The lines should touch the faces.” “The arms are different lengths.” “No fades, he should rise from below.” Small, specific, visual. None of them was a rewrite.

That’s the honest shape: one big spec in, one big build out, then a conversation of corrections, like a director leaning over an editor’s shoulder. If your prompt was a sheet and a set of numbers, the corrections stay small.

What to take from this

If you want an agent to reproduce motion, you need two things most people skip.

1. Give it the whole thing at once. ffmpeg ... tile= and one image. Not a link, not a description, not 700 files.
2. Give it numbers. Pick the two or three quantities that make the motion feel like itself (here: what’s pinned, where it bends, the timing of one cycle), measure them over every frame, hand them over as JSON.

The same trick works on a UI animation you want to match, a product demo you want to rebuild, a motion study for a client. It also works as a way to study someone else’s piece, which is the best reason to do it. You end up understanding why the original works.

Big love, Frank

Frank, thank you for the work, for the long nights of art and near-art talk, and for letting me turn your balancing man into a beer invite. Everything good about the piece is yours. The banana and the trousers are my fault.

See you soon, whatever day it turns out to be. First round is on me.

If you want to learn to set up this kind of loop yourself, that’s exactly what my workshops are for. Request a session and bring a video you can’t stop watching.

Frank on Instagram: @franknitty3000 . Studio and portfolio: franknitty3000.com . The original reel: instagram.com/reel/DcY4nJnz1tk . My version: egor3000.egor.lol .