Skip to main content

Qwen-LiveTranslate-Realtime WebRTC access

Qwen-LiveTranslate-Realtime WebRTC access

WebRTC uses media tracks to transmit audio and DataChannel to transmit model events and text. The web client can use the browser’s native API, while other clients can use libraries that support standard WebRTC. Model Studio does not provide a dedicated WebRTC SDK. This document provides the WebRTC integration procedure for this model, along with a complete browser-side example.

Scope of Application

This document is intended for Qwen-LiveTranslate-Realtime. The existing guidelines list qwen3.8-livetranslate-flash-realtime and qwen3.5-livetranslate-flash-realtime, which have different configurations and event fields; see Client Events for details. The examples in this document use qwen3.8-livetranslate-flash-realtime.
  • Use the API Key and Workspace ID corresponding to your selected region and workspace.
  • Beijing domain: {WorkspaceId}.cn-beijing.maas.aliyuncs.com; Singapore domain: {WorkspaceId}.ap-southeast-1.maas.aliyuncs.com.
  • Long-term API Keys are stored on the server side and should not be included in browser or mobile applications. For instructions on generating temporary API Keys and their permissions, see Generate Temporary API Key.
When capturing audio in the browser, use a secure context; when using the microphone, request microphone permissions. When receiving audio, handle browser autoplay restrictions.

Authentication and SDP Exchange

The browser submits an Offer SDP to the business AppServer. The AppServer forwards the following request using the API Key and returns the Answer SDP unchanged to the browser; long-term API Keys do not reach the frontend.
  • Method: POST
  • Beijing address: https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api/v1/webrtc/realtime?model=qwen3.8-livetranslate-flash-realtime
  • Singapore address: https://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api/v1/webrtc/realtime?model=qwen3.8-livetranslate-flash-realtime
  • Request headers: Authorization: Bearer <API_KEY>, Content-Type: application/sdp
  • Request body: Offer SDP string.
  • Successful response: HTTP 200, with the Answer SDP in the body; failure returns 4xx.
WebRTC performs authentication during the SDP exchange and does not require a prior AOQ Token. In the examples below, /api/realtime/sdp refers to a custom interface built by the business AppServer, which must implement the aforementioned forwarding process—it is not a URL provided by Model Studio.

Workflow

WebRTC connection sequence
  1. Prepare the API key and target model parameters on AppServer.
  2. Create the client connection object and register callbacks.
  3. Configure media directions and DataChannels without attaching upstream media tracks.
  4. Exchange SDP through AppServer and wait for the connection and event channel to become ready.
  5. Configure the model according to its protocol. Attach the required upstream media tracks after confirmation.
  6. Stop capture, close the connection, and release resources when interaction is complete.
  • Audio is transmitted via WebRTC media tracks; the application is responsible for obtaining the local audio track, binding the remote player, and sending/receiving DataChannel messages according to the target model’s event definitions.
  • WebRTC authentication takes place during SDP exchange; no AOQ token is required. Proxy authentication and SDP through AppServer to keep the long-term API key out of frontend code.
  • Connection success indicates that the transport channel has been established, but it does not mean that the model has completed initialization. When using qwen3.8-livetranslate-flash-realtime on this page, send session.update and, after receiving session.updated, enable the required upstream media stream.

Complete Example

The following is a complete browser-side example. First, implement the /api/realtime/sdp API of the business AppServer according to Authentication and SDP Exchange; then open this HTML page from the same origin as the business AppServer. The protocol, host, and port must match. For example, both the page and /api/realtime/sdp are served by http://localhost:3000. The server is responsible for storing the API Key and forwarding the SDP; do not place long-term API Keys in the page. Click “Finish input” to send session.finish, wait for session.finished, and after the remaining audio has finished playing, click “Disconnect after audio playback.” During connection setup or if you need to exit early, click “Cancel and disconnect”; this operation will immediately close the connection and may result in lost translations that have not yet been returned or audio that has not yet been played. This example establishes one connection at a time; reload the page to use it again. 
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>WebRTC access example</title>
</head>
<body>
<button id="start-session" type="button">Start translation</button>
<button id="finish-input" type="button" disabled>Finish input</button>
<button id="end-session" type="button" disabled>Disconnect after audio playback</button>
<button id="cancel-session" type="button" disabled>Cancel and disconnect</button>
<p id="session-status" role="status" aria-live="polite">Click "Start translation" to connect.</p>
<script>
const startSessionButton = document.getElementById("start-session");
const finishInputButton = document.getElementById("finish-input");
const endSessionButton = document.getElementById("end-session");
const cancelSessionButton = document.getElementById("cancel-session");
const sessionStatus = document.getElementById("session-status");
let translationFinished = false;
const pc = new RTCPeerConnection({ iceServers: [ ] });

const channels = new Set();
const senders = new Map();
const remoteAudio = document.createElement("audio");
remoteAudio.controls = true; // Users can play audio manually if autoplay is blocked.
remoteAudio.autoplay = true;
document.body.appendChild(remoteAudio);

let localStream = null;
let eventChannel = null;
let stopped = false;
let started = false;
let sessionCreated = false;
let updateSent = false;
let modelReady = false;
let mediaStarted = false;
let finishRequested = false;
const requestController = new AbortController();
const media = {
  sendAudio: true,
  receiveAudio: true,
};

function mediaDirection(send, receive) {
  if (send && receive) return "sendrecv";
  if (send) return "sendonly";
  if (receive) return "recvonly";
  return "inactive";
}

async function configureMedia() {
  // Keep m=audio. The target service must support the selected directions.
  const transceiver = pc.addTransceiver("audio", {
    direction: mediaDirection(media.sendAudio, media.receiveAudio),
  });
  if (media.sendAudio) senders.set("audio", transceiver.sender);
  if (!media.sendAudio) return;
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: media.sendAudio,
    video: false,
  });
  // The user may end the call while the permission prompt is open.
  if (stopped) {
    stream.getTracks().forEach(track => track.stop());
    throw new Error("Session ended");
  }
  localStream = stream;
  // sender.track is still empty, so no captured data is sent to the model.
}
// See Send and receive events over DataChannels for bindDataChannel.
pc.ondatachannel = ({ channel }) => bindDataChannel(channel);
const clientChannel = pc.createDataChannel("oai-events");
bindDataChannel(clientChannel);
pc.onconnectionstatechange = () => {
  console.log("WebRTC state:", pc.connectionState);
  if (pc.connectionState === "connected") {
    tryInitializeModel();
  } else if (["failed", "disconnected", "closed"].includes(pc.connectionState)) {
    // This example ends the session; implement recovery as needed.
    endSession();
  }
};

function waitForIceComplete() {
  return new Promise((resolve, reject) => {
    const timer = setTimeout(() => finish(new Error("ICE gathering timed out")), 15000);
    function finish(error) {
      clearTimeout(timer);
      pc.removeEventListener("icegatheringstatechange", check);
      requestController.signal.removeEventListener("abort", cancel);
      error ? reject(error) : resolve();
    }
    function check() {
      if (pc.iceGatheringState === "complete") finish();
    }
    function cancel() { finish(new Error("Session ended")); }
    pc.addEventListener("icegatheringstatechange", check);
    requestController.signal.addEventListener("abort", cancel, { once: true });
    if (stopped) cancel();
    else check();
  });
}

function normalizeAnswerSdp(sdp) {
  return String(sdp).trim().replace(/\r?\n/g, "\r\n") + "\r\n";
}

async function startSession() {
  if (started || stopped) return;
  started = true;
  try {
    await configureMedia();
    if (stopped) return;
    const offer = await pc.createOffer();
    await pc.setLocalDescription(offer);
    await waitForIceComplete();

    // Implement this same-origin application endpoint; it is not a Model Studio API.
    // AppServer forwards SDP with the API key and returns the raw Answer SDP.
    const response = await fetch("/api/realtime/sdp", {
      method: "POST",
      headers: { "Content-Type": "application/sdp" },
      body: pc.localDescription.sdp,
      signal: requestController.signal,
    });
    if (!response.ok) {
      throw new Error(`SDP exchange failed:${response.status} ${await response.text()}`);
    }
    const answerSdp = normalizeAnswerSdp(await response.text());
    if (stopped) return;
    await pc.setRemoteDescription({ type: "answer", sdp: answerSdp });
    // Continue through connectionstatechange, DataChannel, and model events.
  } catch (error) {
    if (!stopped) console.error("Connection failed:", error);
    endSession();
  }
}
function tryInitializeModel() {
  if (stopped || updateSent || !sessionCreated ||
      pc.connectionState !== "connected" ||
      eventChannel?.readyState !== "open") return;
  sendModelEvent({
    event_id: `event_${crypto.randomUUID()}`,
    type: "session.update",
    session: {
      "output_modalities": media.receiveAudio ? ["text", "audio"] : ["text"],
      "audio": {
        "input": {
          "turn_detection": {
            "type": "speaker_detection",
            "threshold": 0.5,
          },
        },
      },
      ...(media.receiveAudio ? {
        "voice": "Tina",
      } : {}),
      "translation": {
        "language": "en",
      },
    },
  });
  updateSent = true;
}

async function handleModelEvent(event, channel) {
  if (stopped) return;
  if (event.type === "session.created") {
    sessionCreated = true;
    eventChannel = channel; // Send configuration through the channel that receives session events.
    tryInitializeModel();
  } else if (event.type === "session.updated" && updateSent) {
    modelReady = true;
    await startSendingMedia();
    if (!finishRequested && !stopped) {
      finishInputButton.disabled = false;
      sessionStatus.textContent = "The model is ready. You can start speaking.";
    }
  } else if (event.type === "session.finished") {
    translationFinished = true;
    finishInputButton.disabled = true;
    endSessionButton.disabled = false;
    sessionStatus.textContent = "Translation is complete. Wait for remaining audio to finish before disconnecting.";
    console.log("Translation processing has ended. Wait for remaining audio to finish before releasing resources.");
  } else if (event.type === "error") {
    console.error("Model error:", event.error);
    endSession();
  } else {
    // Display text, transcripts, or response status as defined by the selected model. Receive audio over media tracks.
    console.log("Model event:", event);
  }
}
async function startSendingMedia() {
  if (stopped || !modelReady || mediaStarted) return;
  mediaStarted = true;

  for (const track of localStream?.getTracks() ?? []) {

    if (stopped) return;
    const sender = senders.get(track.kind);
    if (sender) await sender.replaceTrack(track);
  }
}

// Register before startSession(). Tracks without streams are also received correctly.
const remoteAudioStream = new MediaStream();
pc.ontrack = ({ track }) => {
  if (stopped) return;
  if (track.kind === "audio" && media.receiveAudio) {
    remoteAudioStream.addTrack(track);
    remoteAudio.srcObject = remoteAudioStream;
    remoteAudio.play().catch(() => {
      console.info("Autoplay is blocked. Use the audio controls to play.");
    });
  }
};
async function finishTranslation() {
  if (stopped || !modelReady || finishRequested) return;
  finishRequested = true;
  try {
    localStream?.getAudioTracks().forEach(track => track.stop());
    const sender = senders.get("audio");
    if (sender) await sender.replaceTrack(null);
    sendModelEvent({ type: "session.finish" });
  } catch (error) {
    finishRequested = false;
    throw error;
  }
}
function endSession() {
  if (stopped) return;
  stopped = true;
  finishInputButton.disabled = true;
  endSessionButton.disabled = true;
  cancelSessionButton.disabled = true;
  sessionStatus.textContent = "The connection is closed. Reload the page to translate again.";
  modelReady = false;
  sessionCreated = false;
  updateSent = false;
  mediaStarted = false;
  requestController.abort(); // Cancel pending SDP requests and ICE gathering.
  localStream?.getTracks().forEach(track => track.stop());
  localStream = null;
  channels.forEach(channel => channel.close());
  channels.clear();
  eventChannel = null;
  pc.close();
  senders.clear();
  remoteAudioStream.getTracks().forEach(track => track.stop());
  remoteAudio.pause();
  remoteAudio.srcObject = null;
  remoteAudio.remove();
}
function sendModelEvent(event, channel = eventChannel) {
  if (stopped || pc.connectionState !== "connected" ||
      channel?.readyState !== "open") {
    throw new Error("Data channel is not ready");
  }
  channel.send(JSON.stringify(event));
}

function bindDataChannel(channel) {
  channels.add(channel);
  channel.onopen = () => {
    if (stopped) return;
    console.log("DataChannel opened:", channel.label);
    tryInitializeModel();
  };
  channel.onmessage = ({ data }) => {
    if (stopped) return;
    let event;
    try { event = JSON.parse(data); }
    catch (error) {
      console.warn("Cannot parse model event:", error);
      return;
    }
    if (!event || typeof event !== "object") return;
    handleModelEvent(event, channel).catch(error => {
      if (!stopped) console.error("Model event handling failed:", error);
      endSession();
    });
  };
  channel.onerror = error => console.error("DataChannel error:", error);
  channel.onclose = () => {
    channels.delete(channel);
    if (channel === eventChannel) endSession();
  };
  if (channel.readyState === "open") channel.onopen();
}

startSessionButton.addEventListener("click", () => {
  startSessionButton.disabled = true;
  cancelSessionButton.disabled = false;
  sessionStatus.textContent = "Connecting...";
  startSession();
});
finishInputButton.addEventListener("click", () => {
  finishInputButton.disabled = true;
  sessionStatus.textContent = "Finishing input...";
  finishTranslation().then(() => {
    if (!translationFinished && !stopped) {
      sessionStatus.textContent = "Input has ended. Waiting for translation to complete.";
    }
  }).catch(error => {
    if (!stopped) {
      finishInputButton.disabled = false;
      sessionStatus.textContent = "Unable to finish input. Try again.";
    }
    console.error("Failed to finish input:", error);
  });
});
endSessionButton.addEventListener("click", endSession);
cancelSessionButton.addEventListener("click", endSession);
</script>
</body>
</html>

1. Prepare authentication and connection parameters

AppServer configures SDP exchange for the target model, region, and workspace. The following parameters apply to standard Realtime connections. For application-specific parameters, follow the corresponding best practice.
SettingValue
MethodPOST
URLhttps://{endpoint}/api/v1/webrtc/realtime?model={model_name}
Content-Typeapplication/sdp
AuthorizationBearer <API_KEY>
Request bodyOffer SDP
Successful responseAnswer SDP
Complete regional, address, and authentication details can be found in Authentication and SDP Exchange. The business AppServer’s interface is implemented by the business itself and is not part of Model Studio’s SDK API.

2. Initialize the Connection Object and Register Callbacks

On the web side, create a RTCPeerConnection; on other platforms, perform the corresponding library initialization. Before establishing the connection, register the following handlers:
  • Connection state changes: Determine whether the connection succeeded, failed, or was disconnected.
  • DataChannel establishment, opening, messages, and closure: carries model events.
  • Remote media track: receives and plays audio, or handles other media supported by the target application.
The complete example creates the connection object, state variables, and callbacks within the same script. Each call creates a single connection; after ending, reinitialize these objects to make another call. The connection object and state variables are detailed in Complete Example. The complete example uses finishRequested to avoid repeatedly sending the end event. Message listeners should be registered before model initialization to avoid missing session events proactively sent by the server. Do not listen only to channels created by the client: the server will push events through a channel named txt, so you must handle the datachannel callback.

3. Configure media directions and DataChannels

Select media directions

Directions are relative to the client. WebRTC transceivers negotiate sending and receiving: sendrecv is bidirectional, sendonly sends only, and recvonly receives only. Available capabilities also depend on the server's SDP answer and model support.
DirectionConfiguration
pushes audio negotiates audio transmission capabilities and prepares the microphone or an external audio track; attach the track after the model is ready.
pulls audio negotiates audio reception capabilities and binds audio playback in the remote track callback
pushes video negotiates video transmission capabilities based on the selected version and supported scenarios, preparing the camera or an external video track
For example, configure audio uplink when inputting speech; configure audio downlink when voice translation is required. The Offer SDP must include the audio media section m=audio; do not remove this media section simply because the application does not need to play audio.

Media Direction for This Model

Model or applicationSend audioReceive audioSend videoReceive video
Qwen-LiveTranslate-Realtime Enable when voice input is active Enable when voice translation is needed Configure according to the selected version and scenario Does not support enabling
This model outputs text or audio but does not output video. This table describes the media directions needed by the application; it does not guarantee that every combination can be negotiated independently. Configuration depends on the target model and server negotiation capabilities.

Prepare media without sending it

Negotiate the required directions before creating the Offer. For models that require initialization confirmation, reserve sending capabilities without attaching captured tracks. After the model is ready, use replaceTrack on the corresponding sender to start transmission. Register media reception handlers in advance. Audio transmission and reception should be configured separately as needed; do not combine microphone activation, audio playback, and model initialization into a single operation.

Media configuration example

The following uses two-way voice as an example. The two switches in media control audio transmission and reception, respectively. addTransceiver pre-negotiates the direction; the captured track is first stored locally and attached to the sender only after the model is ready. See full example for the code.

Create event channels

Create a DataChannel before creating an Offer so that the SDP includes data channel negotiation. Refer to Data Channel Send and Receive Events for how the client and server handle channel creation. The DataChannel is responsible for sending model initialization, text, and control events, and for receiving model results, status, and errors. The media track is responsible for audio transmission. The complete example creates a client event channel and registers the server-side channel callback before creating an Offer.

4. Establish Connection

Complete the connection in the following order:
  1. Call createOffer and setLocalDescription.
  2. Collect candidate information according to the ICE methods supported by the server; in this example, the local SDP is sent after ICE collection is complete.
  3. Send the local SDP to the business AppServer, which uses the API Key to request the Model Studio SDP exchange interface.
  4. Check the HTTP status, and upon success, call setRemoteDescription using the returned Answer SDP.
  5. Wait for the connection status to become successful, and confirm that the DataChannel used for sending model events has been opened.
SDP exchange success or completion of setRemoteDescription only indicates that the corresponding negotiation step has been completed; it cannot be directly taken as proof that the media connection has succeeded. Model events also cannot be sent when the DataChannel is not in the open state.

Connection Code and Status Callbacks

Listen to connectionstatechange to determine the connection status, and listen to icegatheringstatechange to wait for the complete local SDP. Model initialization also requires the open of the DataChannel and the model’s ready event; it should not rely solely on the completion of SDP exchange for judgment. The full example completes SDP exchange via startSession() and handles the connection result through the connection status callback. The business AppServer should forward the Offer according to the address and request headers specified in Section 1; the sample API is expected to return plain SDP text. Do not pass content containing HTTP response headers, logs, or JSON wrappers directly to setRemoteDescription.

5. Exchange model events and media

Initialize the model

When the connection and event channel are available, initialize the target model according to its protocol. Store server initialization events and combine them with channel state to advance the flow; do not assume a fixed callback arrival order. For example, when configuring a model using session.update, construct the event according to its protocol and wait for configuration confirmation before sending input. Parameters such as voice, output modalities, VAD, and sampling rate should all be derived from the current model’s documentation. session.created indicates session creation; it should not be uniformly regarded as configuration having taken effect. For models that require confirmation of session configuration, wait for session.updated. See Client Events and Server Events for the model’s event definitions. Media transmission and available modes should be handled as described on this page.

Model Event Examples

Audio is transmitted via the media track and does not use WebSocket’s media send event.
The following example demonstrates the session.created → session.update → session.updated process for qwen3.8-livetranslate-flash-realtime. Configure the fields according to the model’s event reference. Only after receiving session.updated confirming that the configuration is complete should you attach the uplink media track. The following configuration translates speech into English using the Tina voice profile. See full example for the code. For the example model qwen3.8-livetranslate-flash-realtime, when audio output is enabled, you can use response.audio.done to check the model’s audio generation status; this event does not indicate that the client has finished playing the audio.

Start media transmission and reception

Once the model is ready, attach the required upstream audio track. Handle the model’s output via the remote track callback and continuously parse the events returned by the DataChannel.
  • Audio travels over RTP media tracks; no input_audio_buffer.append event is required.
  • DataChannels carry text, status, control, and error events.
In the complete example, startSendingMedia() attaches the upstream track, and ontrack processes the received media.

6. End interaction and release resources

For the example model qwen3.8-livetranslate-flash-realtime, after completing audio input, stop capturing and transmitting audio, and send session.finish via DataChannel. Continue receiving the final sentence in its original form, the translation, and the translated audio, and wait for session.finished. This event indicates that the model has finished processing the translation, but it does not mean that the client has finished playing the remaining translated audio; if full playback is required, call endSession() to release resources after playback is complete. In the complete example, finishTranslation() uses the created audio track and DataChannel. If input comes from a file or business buffer, the buffered audio must be sent first before ending the input. The implementation of finishTranslation() can be found in Complete Example. endSession() only releases the connection and media resources; it does not send an end event. After completing the model interaction, the client’s business layer stops the locally captured track held by the application, closes the DataChannel and RTCPeerConnection, releases resources such as the player, and cleans up the business state. Clear model readiness after a failed or unexpected disconnection. Initialize the model again on a new connection instead of reusing the previous connection's session state. In the complete example, endSession() is responsible for releasing the connection and media resources. Call endSession() after completing the model interaction and processing the remaining translated audio. If recording is additionally used by the business, MediaRecorder must also be stopped.

Send and receive events over the Data channel

Check that the channel is open before calling send with a serialized event. Parse incoming messages in its message callback, including events on server-created channels. The names, fields, parameters, and timing of sent events follow the target model’s Client Event Definitions; the parsing method, state changes, results, and error handling after receiving messages adhere to the model’s Server Event Definitions. Initialization events are sent as needed according to the model’s requirements, while subsequent text and control events are sent based on business needs.

DataChannel example

Use the same binding function for client-created and server-created channels. It handles transport and JSON parsing; handleModelEvent in section 5 handles model semantics. Use sendModelEvent for model initialization, text, or control events only when permitted by the model's protocol. In the complete example, sendModelEvent() and bindDataChannel() send and receive events. In the complete example, startSession() is called at the start of translation, finishTranslation() at the end of input, and endSession() after finishing translation and playback cleanup. These functions are examples provided in this document and are not built-in API of WebRTC.

Qwen-LiveTranslate-Realtime Event Definitions

Model or applicationClient events (send)Server events (receive)
Qwen-LiveTranslate-RealtimeClient EventsServer Events
Also follow these rules:
  • Use the type field of events to identify model events, and handle associated identifiers according to this model’s event reference.
  • Operations such as initialization confirmation, input submission, response cancellation, and task completion—required by the model—are all handled within the Data channel according to the corresponding protocol; these constitute the model’s interaction semantics and are distinct from disconnecting the transport connection.
  • Audio data is transmitted over media tracks. When using the model event reference, follow this page for media encapsulation and available interaction modes. Do not directly apply the WebSocket method for sending media.

Event references