bytedance/lipsync/audio-to-video

Bytedance LipSync is an advanced audio-to-video model designed to create lifelike lip movements that perfectly sync with the given audio input. The lip-sync feature can perfectly synchronize the lip movements of characters in videos generated by Bytedance AI with locally recorded or online-generated dubbing/singing files, achieving the effect of real people speaking or singing, and making the video appear lifelike!

Naturally and Highly Matched Lip Movements:
The lip movements of characters in the generated video not only synchronize precisely with the audio but also create unique movement trajectories based on individual facial features and physiological structures, significantly enhancing the video’s naturalness and realism.

Clear Facial Muscle Texture:
The changes in lip movements accurately drive the facial muscles, adjusting in real-time and meticulously presenting the stretching and contraction of muscles during lip movement, resulting in a highly coordinated visual effect that further enhances the overall realism and immersive quality of the video.

Vivid and Lifelike Imagery:
The areas outside the face in the generated video remain consistent with the original video, ensuring the integrity and continuity of the original footage, avoiding interference with non-target areas during the generation process, and restoring the original appearance of the video to the greatest extent possible.

LatentSync: An End-to-End Lip-Sync Method Based on Audio-Conditioned Latent Diffusion Models

We present LatentSync, an end-to-end lip-sync method based on audio-conditioned latent diffusion models without any intermediate motion representation, diverging from previous diffusion-based lip-sync methods based on pixel-space diffusion or two-stage generation. Our framework can leverage the powerful capabilities of Stable Diffusion to directly model complex audio-visual correlations.

LatentSync uses the Whisper to convert melspectrogram into audio embeddings, which are then integrated into the U-Net via cross-attention layers. The reference and masked frames are channel-wise concatenated with noised latents as the input of U-Net. In the training process, we use a one-step method to get estimated clean latents from predicted noises, which are then decoded to obtain the estimated clean frames. The TREPA, LPIPS and SyncNet losses are added in the pixel space.