What is LatentSync and is it good in 2026?

LatentSync has collected about 5,900 stars and 959 forks on GitHub since ByteDance open-sourced it in December 2024, which makes it the most-watched open-source lip sync release of the diffusion era. LatentSync is an open-source model that edits the mouth in an existing video to match new audio, generating the lip region directly with audio-conditioned latent diffusion instead of going through a 3D face model or facial landmarks. The same job runs hosted on sync-3, which lip syncs any footage zero-shot at up to 4K and dubs it across 95+ languages in one pass.

If you are looking up LatentSync, you are usually asking one of three things: what it actually does, whether you can run it yourself, and whether it is still the best open-source option now that development has gone quiet. This page answers all three, with every spec pulled from the repo and the paper as of July 2026.

LatentSync re-syncs the lips in a video you already have

LatentSync takes an existing video of someone talking plus a new audio track, and regenerates the mouth region so the lips match the new audio. Everything else about the footage stays: the performance, the camera move, the background. That puts it in the same family as Wav2Lip, the 2020 model that defined this task, and in a different family from talking head generators like SadTalker, which start from a single photo and invent the whole performance.

The release mattered because of who shipped it and how. ByteDance, the company behind TikTok, published the paper “LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervision” on arXiv in December 2024 and put the full code and weights on GitHub under the Apache 2.0 license, so you can run it commercially for free. For most of 2025 it was the default answer to “what is the best open-source lip sync model,” and it still tops most of those lists today.

How LatentSync works: diffusion in latent space, no 3D middle step

LatentSync generates the synced mouth directly with an audio-conditioned latent diffusion model, skipping the intermediate motion representations older methods relied on. Earlier open-source approaches either warped pixels in 2D (Wav2Lip) or generated 3D face model coefficients first (SadTalker). LatentSync instead runs Stable-Diffusion-style denoising in latent space, with Whisper audio embeddings fed into the U-Net through cross-attention so the sound drives the mouth.

The paper’s real contribution is making that training converge. The authors found diffusion lip sync models suffer from a shortcut problem: the model learns to copy visual patterns from nearby frames and ignores the audio. Their fix is supervising training with SyncNet, an audio-visual sync judge, and they published an empirical study of what makes SyncNet training stable, lifting its accuracy from 91% to 94% on the HDTF test set. A second mechanism, TREPA (Temporal REPresentation Alignment), reduces the frame-to-frame flicker diffusion models are prone to. Those two ideas are why LatentSync clips look noticeably sharper and steadier than Wav2Lip’s blurry mouth crops.

LatentSync 1.5 and 1.6 fixed temporal consistency and resolution

ByteDance shipped two significant upgrades in 2025, and the version you run changes what you get. Per the repo’s release notes:

Version	Released	What changed
1.0	December 2024	Original release, 256px face region
1.5	March 2025	Better temporal consistency, better results on Chinese-language video, inference on 8GB of VRAM
1.6	June 2025	Trained at 512px to cut the blurriness, inference needs 18GB of VRAM

Version 1.6 is the quality pick and 1.5 is the consumer-GPU pick, and that trade is the first real catch: the sharper model roughly doubles the VRAM requirement, which prices out most consumer cards. If you do not want to manage a GPU at all, hosted endpoints for LatentSync exist on services like Replicate and fal, billed per run.

Where LatentSync falls short in 2026

LatentSync has not shipped a release since June 2025, and as of July 2026 the repo’s last commit is over a year old with 228 issues open. It is still genuinely good at its one job, but a few limits show up fast in production use:

• The face region caps at 512 pixels. Even in 1.6, the model generates a cropped face region at 512px that gets composited back into the frame. On a 4K talking head shot, the mouth is upscaled, and the seam between generated and original pixels can show.
• Sync only, nothing around it. LatentSync syncs lips to an audio file you already have. No translation, no text-to-speech, no voice cloning, so an actual dubbing workflow means chaining three or four separate tools before LatentSync runs at the end.
• It wants clean, frontal footage. Like most research models, quality drops on side profiles, occlusions, low light, and multiple speakers in frame, the exact shots real footage is full of.
• You operate it. It is a research codebase with Python dependencies, model checkpoints, and real GPU requirements, and with development quiet for a year, you inherit the open bugs.

None of that is a knock on the research. It is the normal gap between a strong open-source model and a product, the same gap we saw with Wav2Lip, which our founding team built and open-sourced in 2020.

LatentSync vs sync-3

Both models re-sync existing footage, so the comparison comes down to resolution, hard shots, and what happens around the sync step. Specs from each project’s own published sources as of July 2026.

	LatentSync 1.6	sync-3
Input	Existing video + audio	Any video or a single photo, + audio
Output resolution	512px face region, composited	Up to 4K at 60fps, full frame
Hard shots (profiles, low light, multi-speaker)	Degrades	Holds, reads the whole scene
Translation, voice cloning, dubbing	No, audio must be ready	Yes, one pass across 95+ languages
How you run it	Self-hosted, 18GB VRAM, or per-run endpoints	Hosted: web, API, MCP, ComfyUI, Premiere
License / cost	Apache 2.0, free + your GPU	3 free videos/month, then paid

Sources: LatentSync, sync-3.

sync-3 generates from a read of the whole scene rather than a cropped face region, which is why it holds on side profiles, close-ups, and multiple speakers, and why output goes up to 4K at 60fps with no compositing seam. The pipeline LatentSync leaves you to assemble is built in: translation, voice cloning, and lip sync run in one pass across 95+ languages, average processing under 3 minutes. And it covers the photo case too, turning one still into a talking video, which LatentSync does not attempt.

How to choose between LatentSync and a hosted model

Start from what you are shipping and what hardware you have. LatentSync is the right call more often than most research code, but the cases split cleanly:

Your job	Best fit	Why
Research baseline or free experimentation	LatentSync	Apache 2.0, full code and weights, strong quality for open-source
Occasional clips without owning a GPU	LatentSync via Replicate or fal	Per-run pricing, no setup
Footage people will watch at HD or 4K	sync-3	Full-frame generation, no 512px mouth upscale
Dubbing into other languages	sync-3	One pass: translation, voice cloning, lip sync, 95+ languages
Real-world footage: profiles, movement, multiple speakers	sync-3	Whole-scene reasoning instead of a frontal face crop

For the wider open-source field around it, including MuseTalk and Wav2Lip, see the best free and open-source lip sync tools.

Frequently asked questions

If you want the result without the checkpoint downloads, run the same footage through sync-3 in the browser: upload a video and new audio, and get the synced cut back, usually in under 3 minutes, with 3 free videos a month to judge the quality yourself.