Transcription and Dialogue Editing

The dialogue workflow has three steps: transcribe the source, preview changes to its words, then create a lip sync generation using that preview. Transcription and preview jobs are asynchronous. Keep their IDs so your application can resume polling after a disconnect.

Prepare the source

Transcription and dialogue edits require media hosted in Sync storage. For local or externally hosted files, follow Asset Uploads: request an upload URL, upload the bytes, and register the asset. Keep the hosted source video URL and asset ID.

Transcription accepts exactly one of sourceVideoUrl and sourceAudioUrl. Supplying extracted audio avoids server-side extraction. A dialogue edit requires the source video URL and word timings on that video’s timeline. Use the video’s own transcription for this flow.

Transcribe the video

curl https://api.sync.so/v2/transcriptions \
  -H "x-api-key: $SYNC_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"sourceVideoUrl": "https://assets.sync.so/docs/example-video.mp4"}'

Submission returns 201 with a job. Poll GET /v2/transcriptions/{id} using the same API key until status is COMPLETED or FAILED. While the job runs, wait the number of milliseconds in retryAfterMs before each poll; estimate.estimatedFinishAt gives the expected finish time when there is enough history. On completion, save the entire transcript, including segment and word IDs, timing, and speaker information. speakerCount and optional difficulty help describe the source; difficulty is informational.

Each word can include confidence for its recognized text. Review segments with speakerLabelFlags; these warnings do not guarantee that the speaker labels are correct. Preserve speaker IDs and word timings when submitting the transcript.

Word startMs and endMs values are absolute milliseconds from the beginning of the source. Optional request startMs defaults to zero; endMs must be greater than startMs. An explicit transcription window can be at most 60 minutes. maxSourceSeconds, when supplied, is positive and cannot exceed 3,600. Identical source-and-window requests reuse an existing job. Transcription does not charge generation credits.

An endMs past the end of the source is clamped to the source’s end, and the job’s sourceEndMs reports the end that was used. If the source video can’t be measured at submission, the window is kept as sent. A startMs at or past the end of the source video returns 422 with errors[0].path set to startMs, and no job is created.

Preview an edit

Send the returned transcript unchanged and describe the edits separately. An operation with kind: "change" replaces one word by its wordId. An operation with kind: "remove" lists contiguous wordIds. Supply 1–100 operations.

If the voice would mispronounce a replacement, such as a name in a personalized video, add pronunciation to the change operation with a plain-text respelling. The voice speaks the respelling, while the job’s edits and result transcript keep replacement:

{"kind": "change", "wordId": "w_2", "replacement": "Siobhan", "pronunciation": "shivawn"}

Write the respelling like an ordinary word, without hyphens or all-caps syllables: the voice may spell out capital letters one by one, so “shiv-AWN” can come out as “shiv A W N”. Short names can still come out differently between takes, so listen to the preview and create a new one if needed. A respelling is 1–100 characters of letters, spaces, hyphens, apostrophes, and periods, with at least one letter. Phoneme markup is not supported. Punctuation at the end of replacement still shapes the intonation. A remove operation that includes pronunciation is rejected with 400.

The following Python example uses the actual transcript returned by a completed transcription and replaces its first word. Set TRANSCRIPTION_ID to your completed transcription job ID. Review the selected word before applying this example to your own dialogue.

import os
import requests
import uuid
headers = {"x-api-key": os.environ["SYNC_API_KEY"]}
transcription_id = os.environ["TRANSCRIPTION_ID"]
response = requests.get(
    f"https://api.sync.so/v2/transcriptions/{transcription_id}",
    headers=headers,
    timeout=30,
)
response.raise_for_status()
transcription = response.json()
if transcription["status"] != "COMPLETED":
    raise RuntimeError("Wait for the transcription to complete first")
transcript = transcription["transcript"]
word = next(word for segment in transcript["segments"] for word in segment["words"])
# Save this key with the request body and reuse both for retries.
idempotency_key = str(uuid.uuid4())
response = requests.post(
    "https://api.sync.so/v2/dialogue-edits",
    headers={**headers, "Idempotency-Key": idempotency_key},
    json={
        "sourceVideoUrl": transcription["sourceVideoUrl"],
        "transcriptionJobId": transcription_id,
        "transcript": transcript,
        "edits": [{"kind": "change", "wordId": word["id"], "replacement": "Hello"}],
    },
    timeout=30,
)
response.raise_for_status()
print(response.json()["id"])

A new submission returns 201. Retry a timeout or transient error with the same Idempotency-Key and body. An equivalent replay returns 200, the original job, and Idempotency-Replayed: true. Use a new key for a new or revised edit. See Idempotent requests.

Poll GET /v2/dialogue-edits/{id} until COMPLETED, COMPLETED_PARTIAL, or FAILED. Both completed statuses are terminal and can carry playable preview audio. Listen to previewAudioUrl before proceeding. previewDurationMs, resultTranscript, and resultSlots describe the edited timing. previewAssetId is always null. To lip sync the preview, pass the dialogue edit job ID as dialogueEdit: { id } (see below).

Inspect partialDetail before rendering a preview. Its failedSlots entries identify affected resultSlots by index and error code. A reason of background_bed_missing means the preview lacks the source background audio. Listen to the preview and create a revised preview if needed.

To revise a preview, submit another job with the adjusted edits. You can pass the previous job’s voiceId to reuse its cloned voice when it was used by your organization under the same shared or connected ElevenLabs key pool. rerunOfJobId records a previous COMPLETED or COMPLETED_PARTIAL job in your organization. It is informational lineage, not a retry or idempotency key. Hints that do not meet these conditions are ignored.

Lip sync the approved preview

Submit dialogueEdit: { "id": "..." } with exactly one video input. It must be the original video used to create the edit. Use its URL or registered asset ID. Do not add an audio input, image input, segments, or dubParams.

curl https://api.sync.so/v2/generate \
  -H "x-api-key: $SYNC_API_KEY" \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: approved-dialogue-take-001' \
  -d '{
    "model": "lipsync-2",
    "input": [{"type": "video", "url": "https://assets.sync.so/docs/example-video.mp4"}],
    "dialogueEdit": {"id": "REPLACE_WITH_COMPLETED_DIALOGUE_JOB_ID"}
  }'

New previews report both segmentLipsyncEnabled and sectionExpansionEnabled as true. Create a new preview if an older one reports either as false. The output timing follows the edited audio. Continue with ordinary generation polling and webhooks. Generation billing and limits apply.

If sending multipart form data, encode dialogueEdit and input as JSON strings. Uploaded video, audio and image file parts are rejected for this workflow. Uploading the video again is not a substitute for referencing the original source.

Render many previews in one batch

To make many variants of one video, such as one per recipient name, create one preview per variant and render them together with the Batch API. The Batch API requires the Scale plan or higher and approval for your organization. Separately, batch rows with dialogueEdit are in early access and enabled per organization.

1. Create the first preview and wait for it to complete. Pass its voiceId on every other preview of the same video so the voice is cloned only once.
2. Pace preview requests: they are limited to 30 per minute.
3. Listen to a few previews before you submit a large batch. Add pronunciation for names the voice gets wrong.
4. Submit one row per completed preview with the same source video and dialogueEdit: {"id": "..."}. Validate with dry_run first.

See Dialogue edit rows for the row format and refusal codes, and Personalized Video Messaging for a complete script.

Handle refusals and failed jobs

Billing holds can return 402; dialogue output duration is also subject to plan limits. Other transcription job failures include EXTRACTION_FAILED, PROVIDER_FAILED, TIMEOUT, and UNKNOWN. Dialogue job errors include extraction, voice cloning, short voice samples, synthesis, edit/source length, planning, timeout, and unknown failures. Inspect the job’s error.code and error.message before creating another attempt. Polling a failed job does not restart it.