Speaker Selection — API

Speaker selection helps you target the right face when a clip or image contains multiple people. You can either let Sync Labs auto-detect the active speaker (video only) or provide a user-selected point from your UI and forward it via the active_speaker_detection DTO on /v2/generate. For using speaker selection in the web app, see the guide.

When to use what

• Auto-detect (video only): fastest setup; best for single/obvious speaker clips. Set auto_detect: true and skip manual fields.
• Manual selection (video or image): best for multiple people or when you want deterministic control. Provide a reference frame and a point on the speaker’s face, or supply bounding boxes if you already have detections for that frame.

Workflow: selecting a speaker in video

Match transcript speakers to faces

For a completed video transcription, speaker analysis can identify a face for each transcript speaker. It supports video up to 2 minutes long and up to 8 speakers.

1. Read GET /v2/transcriptions/{id}/speaker-analysis. Analysis may already be available. If it has not run, call POST on the same path with the transcription’s revision and sourceVideoUrl. To retry an unavailable result, also send retry: true. Poll the GET endpoint while the status is processing.
2. Choose a speaker whose status is matched. Call POST /v2/transcriptions/{id}/speaker-analysis/face-track-slices with revision, speakerId, startFrame, endFrame, and fps. These frame bounds refer to the full source video; the end frame is excluded.
3. Pass the returned url as the segment’s optionsOverride.active_speaker_detection.bounding_boxes_url. The file contains one bounding box per segment frame. If url is null, the face is absent from that segment; use another selection method or leave it unlipsynced.

For example, a segment from 5 to 10 seconds at 30 fps uses startFrame: 150 and endFrame: 300 in the slice request. Manual frame_number in that segment’s optionsOverride still starts at 0.

An uncertain speaker has no reliable face match. Use the returned reason and ask the user to select a face, or use auto-detect. If the API returns 409 because the revision changed, fetch the current transcription and analysis before retrying. Analysis allows 3 requests per minute; face-track slicing allows 60.

Workflow: selecting a speaker in images (sync-3)

When your image contains multiple faces, you can specify which one to lipsync by providing manual coordinates.

ActiveSpeaker DTO fields

See the full API reference for active_speaker_detection.

• auto_detect (boolean): let Sync Labs pick the active speaker automatically. Defaults to false within an explicit active_speaker_detection object. If the whole object is omitted on a supported video dubbing request, Sync enables auto-detection. Set it to false to disable automatic selection. Manual face selections, speaker mappings, and dialogue edits keep their own settings.
• v3 (boolean, optional): enable ASD v3.
• frame_number (number): zero-based frame index that corresponds to the provided coordinates. At the top level, count from the start of the input video. In a segment’s optionsOverride, count from the start of that segment. For a segment starting at 5 seconds in a 30 fps video, source frame 150 is segment frame 0.
• coordinates ([x, y]): reference point on the speaker’s face in frame_number.
• bounding_boxes (([x1, y1, x2, y2] | [] | null)[], optional): per-frame array of bounding boxes across the video. Each entry corresponds to that frame. Set it to exactly four integers, [x1, y1, x2, y2] (x1,y1 = top-left; x2,y2 = bottom-right), for the detected face. Use null or [] for a frame with no face. Any other entry, such as two or five coordinates or a fractional value, fails the request with a 400 validation error. No generation is created. The same rule applies to bounding_boxes in a segment’s optionsOverride. Use this instead of frame_number + coordinates when you already run detection over the clip.
• bounding_boxes_url (string, optional): URL pointing to a JSON file containing the bounding boxes. Use this instead of inline bounding_boxes to avoid large request payloads. The JSON file must contain a bounding_boxes array with one entry per frame, matching the format above.

Request examples

import { SyncClient } from "@sync.so/sdk";
const sync = new SyncClient();
const response = await sync.generations.create({
  input: [
    { type: "video", url: "https://assets.sync.so/docs/example-video.mp4" },
    { type: "audio", url: "https://assets.sync.so/docs/example-audio.wav" }
  ],
  model: "lipsync-2",
  options: {
    activeSpeakerDetection: {
      autoDetect: false,
      frameNumber: 240,
      coordinates: [640, 360]
    }
  }
});

curl -X POST https://api.sync.so/v2/generate \
  -H "x-api-key: $SYNC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lipsync-2",
    "input": [
      { "type": "video", "url": "https://assets.sync.so/docs/example-video.mp4" },
      { "type": "audio", "url": "https://assets.sync.so/docs/example-audio.wav" }
    ],
    "options": {
      "active_speaker_detection": {
        "auto_detect": false,
        "frame_number": 240,
        "coordinates": [640, 360]
      }
    }
  }'

import { SyncClient } from "@sync.so/sdk";
const sync = new SyncClient();
await sync.generations.create({
  input: [
    { type: "video", url: "https://assets.sync.so/docs/example-video.mp4" },
    { type: "audio", url: "https://assets.sync.so/docs/example-audio.wav" }
  ],
  model: "lipsync-2",
  options: {
    activeSpeakerDetection: {
      autoDetect: false,
      // boundingBoxes aligned to video frames; null where no box is present.
      boundingBoxes: [
        null,                               // frame 0
        [520, 280, 760, 520],               // frame 1 (speaker A) -> [x1,y1,x2,y2]
        [120, 260, 320, 500],               // frame 2 (speaker B)
        null                                // frame 3
        // ...one entry per frame in the clip
      ]
    }
  }
});

import { SyncClient } from "@sync.so/sdk";
const sync = new SyncClient();
await sync.generations.create({
  input: [
    { type: "image", url: "https://assets.sync.so/docs/example-image.jpg" },
    { type: "audio", url: "https://assets.sync.so/docs/example-audio.wav" }
  ],
  model: "sync-3",
  options: {
    activeSpeakerDetection: {
      autoDetect: false,
      frameNumber: 0,
      coordinates: [512, 384]  // center of detected face in image pixels
    }
  }
});

{
  "bounding_boxes": [[520, 280, 760, 520], null, [120, 260, 320, 500], null]
}

import { SyncClient } from "@sync.so/sdk";
const sync = new SyncClient();
await sync.generations.create({
  input: [
    { type: "video", url: "https://assets.sync.so/docs/example-video.mp4" },
    { type: "audio", url: "https://assets.sync.so/docs/example-audio.wav" }
  ],
  model: "lipsync-2",
  options: {
    activeSpeakerDetection: {
      autoDetect: false,
      boundingBoxesUrl: "https://your-cdn.com/bounding-boxes.json"
    }
  }
});