SciGroveBeta
Machine Learning

JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion

Yicheng Xiao, Wenxun Dai, Xinran Qin, Lin Song

Featured August 15, 2026

Original Paper

AI-generated analysis — This is SciGrove's AI interpretation of the paper, not peer-reviewed content. Always refer to the original paper.

Simply

A new AI system lets you edit live video streams in real-time, like adding a sweater to a cat, by cleverly processing video in small chunks and using special training tricks to keep the edits consistent and true to the original video over long periods.

In depth
The paper introduces JoyAI-Video-Edit, an autoregressive diffusion framework for real-time, open-ended video editing. It addresses challenges like train-inference mismatch and temporal drift in streaming video by combining chunk-wise autoregressive adaptation, Source-Anchored Distribution Matching Distillation (SA-DMD), and Long-Horizon Autoregressive Distillation. This allows the model to process video streams causally, maintain source fidelity, and ensure long-term temporal consistency at high frame rates.

Key Takeaways

  • 1
    Achieves real-time, open-ended video editing at 30 FPS for 720p resolution on a single GPU, significantly outperforming existing streaming editors.
  • 2
    Introduces Source-Anchored Distribution Matching Distillation (SA-DMD) to preserve source fidelity and mitigate drift during few-step generation by guiding the teacher model along text-conditioning and source-fidelity axes.
  • 3
    Employs chunk-wise autoregressive adaptation with a sliding temporal window and a global sink, alongside Long-Horizon Autoregressive Distillation, to manage computational resources and maintain long-term temporal consistency in unbounded video streams.

Conceptual Flow

HIGH LEVEL
1
Methodology: Real-Time Streaming Video Editing

The system takes a live video and editing instructions, then processes it piece by piece to create an edited video instantly, without waiting for the whole video.

Live Video Stream
Editing Instructions
Process Chunks & Edit
Edited Video Stream
2
Results: High Quality & Speed

The new method makes edited videos look much better and processes them much faster than older streaming methods, even matching the quality of systems that take a long time.

Old Streaming Editors
New JoyAI-Video-Edit
Compare Quality & Speed
Lower Quality, Slower
Higher Quality, Faster

This breakdown was generated by SciGrove. Get the same analysis — intuition, storyboard, peer review, a runnable prototype and a glossary — on any paper you upload or paste a DOI for.