What is MuseTalk? The real-time lip sync model
What MuseTalk is, how Tencent's real-time model re-syncs the mouth in existing video, what changed in v1.5, where it falls short, and what to use instead.
MuseTalk has pulled in about 6,200 stars and 889 forks on GitHub since Tencent’s Lyra Lab open-sourced it in April 2024, and its headline claim is the one no other open-source lip sync model makes: it runs at 30fps or faster on a single NVIDIA Tesla V100, in real time. MuseTalk is an open-source model that edits the mouth region of an existing video to match new audio, generated by a single-step inpainting pass in the latent space of a variational autoencoder rather than a slow multi-step diffusion loop. The same job runs hosted on sync-3, which lip syncs any footage at up to 4K and dubs it across 95+ languages in one pass.
If you are looking up MuseTalk, you usually want three things answered: what it actually does, whether you can run it yourself, and whether its speed still makes it the right pick in 2026. This page covers all three, with every spec pulled from the repo as of July 2026.
MuseTalk re-syncs the mouth in a video you already have, in real time
MuseTalk takes an existing video of a person talking and a separate audio track, and regenerates the mouth so the lips match the new sound. The rest of the footage is untouched: the head motion, the expression, the background. That puts it in the same task family as Wav2Lip and LatentSync, and in a different family from SadTalker, which starts from one photo and animates a whole head from scratch.
What sets MuseTalk apart inside that family is speed. Most open-source lip sync models trade quality for time and take many seconds per frame; MuseTalk was built to keep up with live playback, hitting 30fps and above on a V100. That makes it the go-to open-source choice for real-time avatars, streaming, and interactive demos where you cannot wait on a render.
How MuseTalk works: single-step latent inpainting, not diffusion
MuseTalk generates the synced mouth by inpainting a masked face region in latent space in a single step, which is the reason it is fast. It encodes each frame with a frozen ft-mse-vae and the audio with a whisper-tiny model, then fuses the audio into the image features through cross-attention inside a U-Net borrowed from stable-diffusion-v1-4. Despite the Stable Diffusion parts, the project is explicit that it is not a diffusion model: there is no iterative denoising, just one inpainting pass over the masked mouth.
That design is the direct trade against a model like LatentSync, which runs a full multi-step latent diffusion loop for sharper output but cannot come close to real time. MuseTalk picks the other end: one pass, live speed, at the cost of some fidelity you would recover with a heavier model.
MuseTalk 1.5 improved lip-sync accuracy and visual quality
Tencent shipped a significant upgrade in 2025, and the version you run changes the output. Per the repo:
| Version | Released | What changed |
|---|---|---|
| 1.0 | April 2024 | Original real-time release, 256x256 face region |
| 1.5 | March 2025 | Perceptual loss, GAN loss, and sync loss, plus a two-stage training strategy and spatio-temporal sampling to balance visual quality against lip-sync accuracy |
Version 1.5 is the one to use. The added losses and the two-stage training close much of the sharpness and sync gap that made 1.0 look clearly synthetic, while keeping the real-time speed that is the whole point of the model.
Where MuseTalk falls short in 2026
MuseTalk is still maintained, with commits into late 2025, but its architecture imposes hard limits that the repo itself is upfront about:
- The face region is 256x256. The model modifies a small, low-resolution mouth crop. To get usable quality on an HD or 4K shot you have to chain a face restorer like GFPGAN as a post-processing step, which adds time and can shift the look of the face.
- Fine facial detail is not preserved. The docs note the original mustache, lip shape, and lip color are not always kept, so the generated mouth can drift from the real person.
- Single-frame generation causes jitter. Because each frame is produced on its own, the mouth can flicker frame to frame, the classic artifact of frame-by-frame methods.
- Sync only, nothing around it. MuseTalk needs an audio file that is already in the target language. No translation, no text-to-speech, no voice cloning, so a real dubbing workflow means stitching three or four tools together with MuseTalk at the end.
- You run it. It is a research codebase with model checkpoints, Python dependencies, and a real GPU requirement to hit its advertised speed.
None of that is a criticism of the work. It is the normal distance between a fast research model and a finished product, the same distance we crossed after our founding team built and open-sourced Wav2Lip in 2020.
MuseTalk vs sync-3
Both re-sync existing footage, so the comparison is about resolution, hard shots, and everything that has to happen around the sync step. Specs from each project’s own published sources as of July 2026.
| MuseTalk 1.5 | sync-3 | |
|---|---|---|
| Input | Existing video + audio | Any video or a single photo, + audio |
| Output resolution | 256x256 face region, composited (GFPGAN to upscale) | Up to 4K at 60fps, full frame |
| Speed | Real time, 30fps+ on a V100 | Under 3 minutes average per video, hosted |
| Hard shots (profiles, low light, multi-speaker) | Degrades | Holds, reads the whole scene |
| Translation, voice cloning, dubbing | No, audio must be ready | Yes, one pass across 95+ languages |
| How you run it | Self-hosted, GPU | Hosted: web, API, MCP, ComfyUI, Premiere |
| License / cost | MIT, free + your GPU | 3 free videos/month, then paid |
sync-3 generates from a read of the entire scene instead of a 256-pixel crop, so it holds on side profiles, close-ups, and multiple speakers, and outputs full frames up to 4K at 60fps with no restorer bolted on afterward. The pipeline MuseTalk leaves you to assemble comes built in: translation, voice cloning, and lip sync run together in one pass across 95+ languages. It also handles the photo case, turning a single still into a talking video, which MuseTalk does not do.
How to choose between MuseTalk and a hosted model
Start from your latency budget and what you are shipping. MuseTalk earns its place when speed is the constraint; the cases split cleanly:
| Your job | Best fit | Why |
|---|---|---|
| Real-time avatar, streaming, or live demo | MuseTalk | Built for 30fps+, the one open-source model that keeps up with playback |
| Research baseline or free experimentation | MuseTalk | MIT license, code and weights free for commercial use |
| Footage people will watch at HD or 4K | sync-3 | Full-frame generation, no 256px mouth crop or restorer pass |
| Dubbing into other languages | sync-3 | One pass: translation, voice cloning, lip sync, 95+ languages |
| Real-world footage: profiles, movement, multiple speakers | sync-3 | Whole-scene reasoning instead of a frontal crop |
For the wider open-source field around it, including Wav2Lip and LatentSync, see the best free and open-source lip sync tools.
Frequently asked questions
What is MuseTalk?
MuseTalk is an open-source real-time lip sync model released by Tencent's Lyra Lab in April 2024. It edits the mouth in an existing video to match new audio using single-step inpainting in the latent space of a VAE, with whisper-tiny audio features driving the generation.
Is MuseTalk really real-time?
Yes. MuseTalk runs at 30fps or faster on a single NVIDIA Tesla V100, which is fast enough for live playback. That real-time speed comes from a single-step inpainting pass rather than the multi-step diffusion loop slower models use.
Is MuseTalk free for commercial use?
Yes. MuseTalk's code is released under the MIT License and the trained models are available for any purpose, including commercial use. You supply the GPU and the setup.
What is the difference between MuseTalk and LatentSync?
Both re-sync the mouth in existing video, but MuseTalk uses one-step latent inpainting for real-time speed at a 256-pixel face region, while LatentSync runs multi-step latent diffusion for sharper output that is far slower. MuseTalk is the speed pick, LatentSync the quality pick among open-source options.
What is the best MuseTalk alternative in 2026?
sync-3 covers the same job hosted: it lip syncs any footage, holds on side profiles and multiple speakers, outputs up to 4K at 60fps instead of a 256-pixel crop, and adds translation and voice cloning in one pass across 95+ languages, with no GPU to manage.
See how sync-3 syncs the full frame and keeps the speaker’s own voice across languages, no checkpoints or restorers to wire up, with 3 free videos a month to judge the quality yourself.