Video Dubbing API Guide

Video dubbing combines translated audio with lip-synced video so dubbed content looks natural in the target language. Sync handles the lipsync step — you provide the video and translated audio, and the API generates matching lip movements.

Prerequisites

• A Sync API key
• A source video (URL or uploaded asset)
• Translated audio in the target language (from a TTS service or human voice actor)

Install the SDK for your language:

$
# Python
$
pip install syncsdk
$
$
# TypeScript
$
npm i @sync.so/sdk

Set your API key:

$
export SYNC_API_KEY="your-api-key"

Basic Dubbing Pipeline

1
import { SyncClient } from "@sync.so/sdk";
2
3
const sync = new SyncClient();
4
5
// Source video with original language
6
const videoUrl = "https://your-cdn.com/original-video.mp4";
7
// Translated audio in target language
8
const dubbedAudioUrl = "https://your-cdn.com/translated-audio-spanish.wav";
9
10
const response = await sync.generations.create({
11
    input: [
12
        { type: "video", url: videoUrl },
13
        { type: "audio", url: dubbedAudioUrl },
14
    ],
15
    model: "lipsync-2",
16
    options: { sync_mode: "cut_off" },
17
});
18
19
const jobId = response.id;
20
console.log(`Dubbing job submitted: ${jobId}`);

1
let generation = await sync.generations.get(jobId);
2
while (!["COMPLETED", "FAILED", "REJECTED"].includes(generation.status)) {
3
    console.log(`Status: ${generation.status}`);
4
    await new Promise((r) => setTimeout(r, 10000));
5
    generation = await sync.generations.get(jobId);
6
}
7
8
if (generation.status === "COMPLETED") {
9
    console.log(`Dubbed video ready: ${generation.outputUrl}`);
10
} else {
11
    console.log(`Dubbing failed for job ${jobId}`);
12
}

Using the ElevenLabs Integration

Sync has a built-in ElevenLabs integration that handles text-to-speech and lipsync in a single API call. Instead of generating audio separately, you pass the translated text directly.

1
import { SyncClient } from "@sync.so/sdk";
2
3
const sync = new SyncClient();
4
5
const response = await sync.generations.create({
6
    input: [
7
        {
8
            type: "video",
9
            url: "https://your-cdn.com/original-video.mp4",
10
        },
11
        {
12
            type: "text",
13
            provider: {
14
                name: "elevenlabs",
15
                voiceId: "EXAVITQu4vr4xnSDxMaL",
16
                script: "Hola, bienvenidos a nuestra plataforma. Hoy les mostraremos las nuevas funciones.",
17
                stability: 0.5,
18
                similarityBoost: 0.75,
19
            },
20
        },
21
    ],
22
    model: "lipsync-2",
23
    options: { sync_mode: "cut_off" },
24
});
25
26
console.log(`Job ID: ${response.id}`);

Supported Languages

Sync’s lipsync models are language-agnostic. They work with audio in any language — the models analyze mouth shapes from the audio waveform, not the language itself. If your translated audio is clear and well-produced, the lipsync output will match.

For a complete translation pipeline walkthrough (transcription, translation, TTS, and lipsync), see the Video Translation API Guide.

Multi-Speaker Dubbing

For videos with multiple speakers, use the segments API to assign different audio tracks to different time ranges. Each segment can reference a separate audio input with a distinct voice.

1
from sync import Sync
2
from sync.common import Audio, Video
3
4
sync = Sync()
5
6
response = sync.generations.create(
7
    input=[
8
        Video(url="https://your-cdn.com/interview.mp4"),
9
        Audio(url="https://your-cdn.com/speaker-a-spanish.wav", ref_id="speaker_a"),
10
        Audio(url="https://your-cdn.com/speaker-b-spanish.wav", ref_id="speaker_b"),
11
    ],
12
    segments=[
13
        {"startTime": 0, "endTime": 15, "audioInput": {"refId": "speaker_a"}},
14
        {"startTime": 15, "endTime": 30, "audioInput": {"refId": "speaker_b"}},
15
    ],
16
    model="lipsync-2",
17
)

See the Segments Guide for full documentation and more examples.

Performance Tips

Next Steps

• Video Translation API Guide — Build a full transcription-to-lipsync pipeline
• Batch API — Process hundreds of dubbing jobs at once
• Segments Guide — Handle multi-speaker videos
• Text-to-Speech Lip Sync Guide — Combine TTS with lipsync