Transcription and Dialogue Editing

The dialogue workflow has three steps: transcribe the source, preview changes to its words, then create a lip sync generation using that preview. Transcription and preview jobs are asynchronous. Keep their IDs so your application can resume polling after a disconnect.

Prepare the source

Transcription and dialogue edits require media hosted in Sync storage. For local or externally hosted files, follow Asset Uploads: request an upload URL, upload the bytes, and register the asset. Keep the hosted source video URL and asset ID.

Transcription accepts exactly one of sourceVideoUrl and sourceAudioUrl. Supplying extracted audio avoids server-side extraction. A dialogue edit requires the source video URL and word timings on that video’s timeline. Use the video’s own transcription for this flow.

Transcribe the video

curl https://api.sync.so/v2/transcriptions \
  -H "x-api-key: $SYNC_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"sourceVideoUrl": "https://assets.sync.so/docs/example-video.mp4"}'

Submission returns 201 with a job. Poll GET /v2/transcriptions/{id} using the same API key until status is COMPLETED or FAILED. Use backoff between requests. On completion, save the entire transcript, including segment and word IDs, timing, and speaker information. speakerCount and optional difficulty help describe the source; difficulty is informational.

Word startMs and endMs values are absolute milliseconds from the beginning of the source. Optional request startMs defaults to zero; endMs must be greater than startMs. An explicit transcription window can be at most 60 minutes. maxSourceSeconds, when supplied, is positive and cannot exceed 3,600. Identical source-and-window requests reuse an existing job. Transcription does not charge generation credits.

Preview an edit

Send the returned transcript unchanged and describe the edits separately. An operation with kind: "change" replaces one word by its wordId. An operation with kind: "remove" lists contiguous wordIds. Supply 1–100 operations.

A change can also carry optional pronunciation: a plain-text respelling the voice speaks instead of replacement, for names it says wrong. For example, {"kind": "change", "wordId": "w_2", "replacement": "Siobhan", "pronunciation": "shivawn"} speaks “shivawn” while the transcript and results keep “Siobhan”. Write the respelling like an ordinary word, without hyphens or all-caps syllables: the voice may spell out capital letters one by one, so “shiv-AWN” can come out as “shiv A W N”. Short names can still come out differently between takes, so listen to the preview and create a new one if needed.

The following Python example uses the actual transcript returned by a completed transcription and replaces its first word. Set TRANSCRIPTION_ID to your completed transcription job ID. Review the selected word before applying this example to your own dialogue.

import os
import requests
headers = {"x-api-key": os.environ["SYNC_API_KEY"]}
transcription_id = os.environ["TRANSCRIPTION_ID"]
response = requests.get(
    f"https://api.sync.so/v2/transcriptions/{transcription_id}",
    headers=headers,
    timeout=30,
)
response.raise_for_status()
transcription = response.json()
if transcription["status"] != "COMPLETED":
    raise RuntimeError("Wait for the transcription to complete first")
transcript = transcription["transcript"]
word = next(word for segment in transcript["segments"] for word in segment["words"])
response = requests.post(
    "https://api.sync.so/v2/dialogue-edits",
    headers=headers,
    json={
        "sourceVideoUrl": transcription["sourceVideoUrl"],
        "transcript": transcript,
        "edits": [{"kind": "change", "wordId": word["id"], "replacement": "Hello"}],
    },
    timeout=30,
)
response.raise_for_status()
print(response.json()["id"])

Submission returns 201. Poll GET /v2/dialogue-edits/{id} until COMPLETED, COMPLETED_PARTIAL, or FAILED. Both completed statuses are terminal and can carry playable preview audio. Listen to previewAudioUrl before proceeding. previewDurationMs, resultTranscript, and resultSlots describe the edited timing. previewAssetId is always null. To lip sync the preview, pass the dialogue edit job ID as dialogueEdit: { id } (see below).

To revise a preview, submit another job with the adjusted edits. You can pass the previous job’s voiceId to reuse its cloned voice when it was used by your organization under the same shared or connected ElevenLabs key pool. rerunOfJobId records a previous COMPLETED or COMPLETED_PARTIAL job in your organization. It is informational lineage, not a retry or idempotency key. Hints that do not meet these conditions are ignored.

Lip sync the approved preview

Submit dialogueEdit: { "id": "..." } with exactly one video input. It must be the original video used to create the edit. Use its URL or registered asset ID. Do not add an audio input, image input, segments, or dubParams.

curl https://api.sync.so/v2/generate \
  -H "x-api-key: $SYNC_API_KEY" \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: approved-dialogue-take-001' \
  -d '{
    "model": "lipsync-2",
    "input": [{"type": "video", "url": "https://assets.sync.so/docs/example-video.mp4"}],
    "dialogueEdit": {"id": "REPLACE_WITH_COMPLETED_DIALOGUE_JOB_ID"}
  }'

The completed preview must report both segmentLipsyncEnabled and sectionExpansionEnabled as true. Continue with ordinary generation polling and webhooks. Generation billing and limits apply.

If sending multipart form data, encode dialogueEdit and input as JSON strings. Uploaded video, audio and image file parts are rejected for this workflow. Uploading the video again is not a substitute for referencing the original source.

Render many previews in one batch

To make many variants of one video, such as one per recipient name, create one preview per variant and render them together with the Batch API. Batch rows with dialogueEdit are in early access and enabled per organization.

1. Create the first preview and wait for it to complete. Pass its voiceId on every other preview of the same video so the voice is cloned only once.
2. Pace preview requests: they are limited to 30 per minute.
3. Listen to a few previews before you submit a large batch. Add pronunciation for names the voice gets wrong.
4. Submit one row per completed preview with the same source video and dialogueEdit: {"id": "..."}. Validate with dry_run first.

See Dialogue edit rows for the row format and refusal codes, and Personalized Video Messaging for a complete script.

Handle refusals and failed jobs

Billing holds can return 402; dialogue output duration is also subject to plan limits. Other transcription job failures include EXTRACTION_FAILED, PROVIDER_FAILED, TIMEOUT, and UNKNOWN. Dialogue job errors include extraction, voice cloning, short voice samples, synthesis, edit/source length, planning, timeout, and unknown failures. Inspect the job’s error.code and error.message before creating another attempt. Polling a failed job does not restart it.