AI system learns to dub videos by watching lips move

Researchers built a machine learning system that can automatically add synthetic speech to videos while matching the speaker's lip movements and emotional tone. This makes it easier to create dubbed versions of films and videos without hiring voice actors or spending time on manual synchronization.

Video dubbing has been expensive and labor-intensive because matching speech to lip movements requires either manual frame-by-frame adjustment or training separate AI systems from scratch — this approach reuses existing speech synthesis models and handles synchronization automatically, which could reduce production costs and speed up localization for films, documentaries, and assistive content.