Tech / Season 2 / EP03
Video Inpainting
It's ruined! The shot was perfect! The light was perfect! Now there's a human train wreck in the middle of my sunset!
The video isn't ruined. It just has a data anomaly. You can patch it.
Patch it? Kurumi, it's a video. I'd have to be a professional VFX artist at Pixar to fix this.
You don't need Pixar. You just need a better algorithm for hallucinating reality.
How? It's... perfect. It's not a blur. It's a wave. A moving, shimmering wave.
This is far more complex than making a photo sing. This is digital surgery.
When you drew that circle, you gave the AI a hint. The segmentation model then perfectly identified and tracked him across every frame, creating a Chad-shaped hole in the timeline.
Okay, so it cut a hole. But how did it know what to fill the hole with?
The AI doesn't just look at the pixels *around* the hole in one frame. That would be flat and fake.
It analyzes the **motion vectors** of the surrounding reality. It learns the speed, direction, and rhythm of the water. Watch—every flicker of surf, every disturbance, is mapped and predicted.
It also creates a **Depth Map**. It knows the breaking waves are 'behind' the man, while the dry, untouched sand is 'in front.' It's reconstructing a 3D simulation of the scene's physics, layer by layer.
So it understands how the scene is *supposed* to behave.
Precisely. Now for the magic.
It uses a **temporally consistent diffusion model**. It hallucinates new wave pixels, frame by frame, ensuring the new water moves in perfect sync with the real water around it—not a single flicker or wobble across the timeline.
It's not just filling a gap. It's re-rendering the scene with the correct lighting, the correct motion, and the correct depth. It's a VFX shot that would have taken a human artist hours, executed in seconds.
And it has to do that... for every single frame?
30 times per second. Each frame is a unique painting that has to be consistent with the one before and the one after. That is the computational miracle.
Incredible. Now I don't have to do a double take!
Yes. You've successfully erased the inconvenient human from your memory.
...hmm, that doesn't sound too right.
