Skip to main content

Qwen-Omni-Realtime WebRTC access

Qwen-Omni-Realtime WebRTC access

WebRTC carries audio and video on media tracks and model events and text over DataChannels. Web applications can use native browser APIs; other platforms can use standard WebRTC libraries. Model Studio does not provide a dedicated WebRTC SDK. This document provides the WebRTC integration procedure for this model, along with a complete browser-side example.

Scope of Application

This document is aimed at Qwen-Omni-Realtime. Existing integration documentation covers: qwen3.8-omni-flash-realtime, qwen3.5-omni-plus-realtime, qwen3.5-omni-flash-realtime. The example model is qwen3.8-omni-flash-realtime.
  • Use the API Key and Workspace ID corresponding to your selected region and workspace.
  • Beijing domain: {WorkspaceId}.cn-beijing.maas.aliyuncs.com; Singapore domain: {WorkspaceId}.ap-southeast-1.maas.aliyuncs.com.
  • Long-term API Keys are stored on the server side and should not be included in browser or mobile applications. For instructions on generating temporary API Keys and their permissions, see Generate Temporary API Key.
The browser uses a secure context when capturing media; request the corresponding permissions when using the microphone or camera. When receiving audio, handle the browser’s autoplay restrictions.

Authentication and SDP Exchange

The browser submits an Offer SDP to the business AppServer. The AppServer forwards the following request using the API Key and returns the Answer SDP unchanged to the browser; long-term API Keys do not reach the frontend.
  • Method: POST
  • Beijing address: https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api/v1/webrtc/realtime?model=qwen3.8-omni-flash-realtime
  • Singapore address: https://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api/v1/webrtc/realtime?model=qwen3.8-omni-flash-realtime
  • Request headers: Authorization: Bearer <API_KEY>, Content-Type: application/sdp
  • Request body: Offer SDP string.
  • Successful response: HTTP 200, with the Answer SDP in the body; failure returns 4xx.
WebRTC performs authentication during the SDP exchange and does not require a prior AOQ Token. In the examples below, /api/realtime/sdp refers to a custom interface built by the business AppServer, which must implement the aforementioned forwarding process—it is not a URL provided by Model Studio.

Workflow

WebRTC connection sequence
  1. Prepare the API key and target model parameters on AppServer.
  2. Create the client connection object and register callbacks.
  3. Configure media directions and DataChannels without attaching upstream media tracks.
  4. Exchange SDP through AppServer and wait for the connection and event channel to become ready.
  5. Configure the model according to its protocol. Attach the required upstream media tracks after confirmation.
  6. Stop capture, close the connection, and release resources when interaction is complete.
  • WebRTC media tracks carry audio and video. Your application obtains local media tracks, attaches remote players, and exchanges DataChannel messages according to the target model's event definitions.
  • WebRTC authentication takes place during SDP exchange; no AOQ token is required. Proxy authentication and SDP through AppServer to keep the long-term API key out of frontend code.
  • A successful connection indicates that the transmission channel has been established, but it does not mean that the model has finished initializing. When using qwen3.8-omni-flash-realtime on this page, start the required upstream media only after sending session.update and receiving session.updated.

Complete Example

The following is a complete browser-side example. First, implement the /api/realtime/sdp interface of the business AppServer as described in Authentication and SDP Exchange; then open this HTML page from the same origin as the business AppServer, ensuring that the protocol, host, and port match. For example, both the page and /api/realtime/sdp are served by http://localhost:3000. The server is responsible for storing the API Key and forwarding the SDP; do not include long-term API Keys in the page. Click “Start” to establish the connection; click “End” to release resources when the interaction concludes. This example establishes one connection at a time; reload the page to initiate another session. 
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>WebRTC access example</title>
</head>
<body>
<button id="start-session" type="button">Start</button>
<button id="end-session" type="button">End</button>
<script>
const pc = new RTCPeerConnection({ iceServers: [ ] });

const channels = new Set();
const senders = new Map();
const remoteAudio = document.createElement("audio");
remoteAudio.controls = true; // Allow manual playback if autoplay is blocked.
remoteAudio.autoplay = true;
document.body.appendChild(remoteAudio);

let localStream = null;
let eventChannel = null;
let stopped = false;
let started = false;
let sessionCreated = false;
let updateSent = false;
let modelReady = false;
let mediaStarted = false;
const requestController = new AbortController();
const media = {
  sendAudio: true,
  receiveAudio: true,
  sendVideo: false,
};

function mediaDirection(send, receive) {
  if (send && receive) return "sendrecv";
  if (send) return "sendonly";
  if (receive) return "recvonly";
  return "inactive";
}

async function configureMedia() {
  for (const kind of ["audio", "video"]) {
    const send = kind === "audio" ? media.sendAudio : media.sendVideo;
    const receive = kind === "audio" ? media.receiveAudio : false;
    // Keep m=audio. The target service must support the selected directions.
    if (kind === "audio" || send || receive) {
      const transceiver = pc.addTransceiver(kind, {
        direction: mediaDirection(send, receive),
      });
      if (send) senders.set(kind, transceiver.sender);
    }
  }
  if (!media.sendAudio && !media.sendVideo) return;
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: media.sendAudio,
    video: media.sendVideo ? {
      width: { ideal: 640 }, height: { ideal: 480 },
      frameRate: { ideal: 2, max: 2 },
    } : false,
  });
  // The user may end the call while the permission prompt is open.
  if (stopped) {
    stream.getTracks().forEach(track => track.stop());
    throw new Error("Session ended");
  }
  localStream = stream;
  // sender.track is still empty, so no captured data is sent to the model.
}
// See Send and receive events over DataChannels for bindDataChannel.
pc.ondatachannel = ({ channel }) => bindDataChannel(channel);
const clientChannel = pc.createDataChannel("oai-events");
bindDataChannel(clientChannel);
pc.onconnectionstatechange = () => {
  console.log("WebRTC state:", pc.connectionState);
  if (pc.connectionState === "connected") {
    tryInitializeModel();
  } else if (["failed", "disconnected", "closed"].includes(pc.connectionState)) {
    // This example ends the session; implement recovery as needed.
    endSession();
  }
};

function waitForIceComplete() {
  return new Promise((resolve, reject) => {
    const timer = setTimeout(() => finish(new Error("ICE gathering timed out")), 15000);
    function finish(error) {
      clearTimeout(timer);
      pc.removeEventListener("icegatheringstatechange", check);
      requestController.signal.removeEventListener("abort", cancel);
      error ? reject(error) : resolve();
    }
    function check() {
      if (pc.iceGatheringState === "complete") finish();
    }
    function cancel() { finish(new Error("Session ended")); }
    pc.addEventListener("icegatheringstatechange", check);
    requestController.signal.addEventListener("abort", cancel, { once: true });
    if (stopped) cancel();
    else check();
  });
}

function normalizeAnswerSdp(sdp) {
  return String(sdp).trim().replace(/\r?\n/g, "\r\n") + "\r\n";
}

async function startSession() {
  if (started || stopped) return;
  started = true;
  try {
    await configureMedia();
    if (stopped) return;
    const offer = await pc.createOffer();
    await pc.setLocalDescription(offer);
    await waitForIceComplete();

    // Implement this same-origin application endpoint; it is not a Model Studio API.
    // AppServer forwards SDP with the API key and returns the raw Answer SDP.
    const response = await fetch("/api/realtime/sdp", {
      method: "POST",
      headers: { "Content-Type": "application/sdp" },
      body: pc.localDescription.sdp,
      signal: requestController.signal,
    });
    if (!response.ok) {
      throw new Error(`SDP exchange failed:${response.status} ${await response.text()}`);
    }
    const answerSdp = normalizeAnswerSdp(await response.text());
    if (stopped) return;
    await pc.setRemoteDescription({ type: "answer", sdp: answerSdp });
    // Continue through connectionstatechange, DataChannel, and model events.
  } catch (error) {
    if (!stopped) console.error("Connection failed:", error);
    endSession();
  }
}
function tryInitializeModel() {
  if (stopped || updateSent || !sessionCreated ||
      pc.connectionState !== "connected" ||
      eventChannel?.readyState !== "open") return;
  sendModelEvent({
    event_id: `event_${crypto.randomUUID()}`,
    type: "session.update",
    session: {
      "modalities": media.receiveAudio ? ["text", "audio"] : ["text"],
      "voice": "Tina",
      "turn_detection": { "type": "server_vad" },
    },
  });
  updateSent = true;
}

async function handleModelEvent(event, channel) {
  if (stopped) return;
  if (event.type === "session.created") {
    sessionCreated = true;
    eventChannel = channel; // Send configuration through the channel that receives session events.
    tryInitializeModel();
  } else if (event.type === "session.updated" && updateSent) {
    modelReady = true;
    await startSendingMedia();
  } else if (event.type === "error") {
    console.error("Model error:", event.error);
    endSession();
  } else {
    // Display text, transcripts, or response status as defined by the selected model. Receive audio over media tracks.
    console.log("Model event:", event);
  }
}
async function startSendingMedia() {
  if (stopped || !modelReady || mediaStarted) return;
  mediaStarted = true;

  for (const track of localStream?.getTracks() ?? []) {

    if (stopped) return;
    const sender = senders.get(track.kind);
    if (sender) await sender.replaceTrack(track);
  }
}

// Register before startSession(). Tracks without streams are also received correctly.
const remoteAudioStream = new MediaStream();
pc.ontrack = ({ track }) => {
  if (stopped) return;
  if (track.kind === "audio" && media.receiveAudio) {
    remoteAudioStream.addTrack(track);
    remoteAudio.srcObject = remoteAudioStream;
    remoteAudio.play().catch(() => {
      console.info("Autoplay is blocked. Use the audio controls to play.");
    });
  }
};
function endSession() {
  if (stopped) return;
  stopped = true;
  modelReady = false;
  sessionCreated = false;
  updateSent = false;
  mediaStarted = false;
  requestController.abort(); // Cancel pending SDP requests and ICE gathering.
  localStream?.getTracks().forEach(track => track.stop());
  localStream = null;
  channels.forEach(channel => channel.close());
  channels.clear();
  eventChannel = null;
  pc.close();
  senders.clear();
  remoteAudioStream.getTracks().forEach(track => track.stop());
  remoteAudio.pause();
  remoteAudio.srcObject = null;
  remoteAudio.remove();
}
function sendModelEvent(event, channel = eventChannel) {
  if (stopped || pc.connectionState !== "connected" ||
      channel?.readyState !== "open") {
    throw new Error("Data channel is not ready");
  }
  channel.send(JSON.stringify(event));
}

function bindDataChannel(channel) {
  channels.add(channel);
  channel.onopen = () => {
    if (stopped) return;
    console.log("DataChannel opened:", channel.label);
    tryInitializeModel();
  };
  channel.onmessage = ({ data }) => {
    if (stopped) return;
    let event;
    try { event = JSON.parse(data); }
    catch (error) {
      console.warn("Cannot parse model event:", error);
      return;
    }
    if (!event || typeof event !== "object") return;
    handleModelEvent(event, channel).catch(error => {
      if (!stopped) console.error("Model event handling failed:", error);
      endSession();
    });
  };
  channel.onerror = error => console.error("DataChannel error:", error);
  channel.onclose = () => {
    channels.delete(channel);
    if (channel === eventChannel) endSession();
  };
  if (channel.readyState === "open") channel.onopen();
}

document.getElementById("start-session").addEventListener("click", startSession);
document.getElementById("end-session").addEventListener("click", endSession);
</script>
</body>
</html>

1. Prepare authentication and connection parameters

AppServer configures SDP exchange for the target model, region, and workspace. The following parameters apply to standard Realtime connections. For application-specific parameters, follow the corresponding best practice.
SettingValue
MethodPOST
URLhttps://{endpoint}/api/v1/webrtc/realtime?model={model_name}
Content-Typeapplication/sdp
AuthorizationBearer <API_KEY>
Request bodyOffer SDP
Successful responseAnswer SDP
Complete regional, address, and authentication details can be found in Authentication and SDP Exchange. The business AppServer’s interface is implemented by the business itself and is not part of Model Studio’s SDK API.

2. Initialize the Connection Object and Register Callbacks

On the web side, create a RTCPeerConnection; on other platforms, perform the corresponding library initialization. Before establishing the connection, register the following handlers:
  • Connection state changes: Determine whether the connection succeeded, failed, or was disconnected.
  • DataChannel establishment, opening, messages, and closure: carries model events.
  • Remote media track: receives and plays audio, or handles other media supported by the target application.
The complete example creates the connection object, state variables, and callbacks within the same script. Each call creates a single connection; after ending, reinitialize these objects to make another call. The connection object and state variables are detailed in Complete Example. Message listeners should be registered before model initialization to avoid missing session events proactively sent by the server. Do not listen only to channels created by the client: the server will push events through a channel named txt, so you must handle the datachannel callback.

3. Configure media directions and DataChannels

Select media directions

Directions are relative to the client. WebRTC transceivers negotiate sending and receiving: sendrecv is bidirectional, sendonly sends only, and recvonly receives only. Available capabilities also depend on the server's SDP answer and model support.
DirectionConfiguration
Push audioNegotiate audio sending capabilities and prepare the microphone or an external audio track; attach the track after the model is ready.
Pull audioNegotiate audio receiving capabilities and bind audio playback in the remote track callback
Push videoWhen the target model supports visual input, negotiate video sending capabilities and prepare the camera or an external video track
Voice conversations typically require bidirectional audio; visual input additionally requires upstream video. The Offer SDP must include an m=audio media section. Do not remove it merely because the application does not need audio playback.

Media Direction for This Model

Model or applicationSend audioReceive audioSend videoReceive video
Qwen-Omni-Realtimeenable when voice input is requiredenable when voice response is neededenable when visual input is requiredcannot enable
This table describes the media directions needed by the application; it does not guarantee that every combination can be negotiated independently. Configuration depends on the target model and server negotiation capabilities.

Prepare media without sending it

Negotiate the required directions before creating the Offer. For models that require initialization confirmation, reserve sending capabilities without attaching captured tracks. After the model is ready, use replaceTrack on the corresponding sender to start transmission. Register media reception handlers in advance. Audio sending, audio receiving, and video sending should be configured separately as needed; do not combine microphone activation, audio playback, and model initialization into a single operation.

Media configuration example

The following example demonstrates two-way voice with optional video upload. The three switches in media control audio sending, audio receiving, and video sending, respectively. addTransceiver pre-negotiates the direction; captured tracks are first stored locally until the model is ready, then mounted to the sender. See full example for the code. Set video resolution and frame rate according to the target model. If local preview and upstream frame rates need to differ, see the Canvas implementation in the best practice.

Create event channels

Create a DataChannel before creating an Offer so that the SDP includes data channel negotiation. Refer to Data Channel Send and Receive Events for how the client and server handle channel creation. DataChannels carry model initialization, text, and control events, and return model results, status, and errors. Media tracks carry audio and video. The complete example creates a client event channel and registers the server-side channel callback before creating an Offer.

4. Establish Connection

Complete the connection in the following order:
  1. Call createOffer and setLocalDescription.
  2. Collect candidate information according to the ICE methods supported by the server; in this example, the local SDP is sent after ICE collection is complete.
  3. Send the local SDP to the business AppServer, which uses the API Key to request the Model Studio SDP exchange interface.
  4. Check the HTTP status, and upon success, call setRemoteDescription using the returned Answer SDP.
  5. Wait for the connection status to become successful, and confirm that the DataChannel used for sending model events has been opened.
SDP exchange success or completion of setRemoteDescription only indicates that the corresponding negotiation step has been completed; it cannot be directly taken as proof that the media connection has succeeded. Model events also cannot be sent when the DataChannel is not in the open state.

Connection Code and Status Callbacks

Listen to connectionstatechange to determine the connection status, and listen to icegatheringstatechange to wait for the complete local SDP. Model initialization also requires the open of the DataChannel and the model’s ready event; it should not rely solely on the completion of SDP exchange for judgment. The full example completes SDP exchange via startSession() and handles the connection result through the connection status callback. The business AppServer should forward the Offer according to the address and request headers specified in Section 1; the sample API is expected to return plain SDP text. Do not pass content containing HTTP response headers, logs, or JSON wrappers directly to setRemoteDescription.

5. Exchange model events and media

Initialize the model

When the connection and event channel are available, initialize the target model according to its protocol. Store server initialization events and combine them with channel state to advance the flow; do not assume a fixed callback arrival order. For example, when configuring a model using session.update, construct the event according to its protocol and wait for configuration confirmation before sending input. Parameters such as voice, output modalities, VAD, and sampling rate should all be derived from the current model’s documentation. session.created indicates session creation; it should not be uniformly regarded as configuration having taken effect. For models that require confirmation of session configuration, wait for session.updated. Model event definitions can be found in Client Events and Server Events. Media transmission and available modes should be handled according to the instructions on this page. When Qwen-Omni-Realtime is accessed over WebRTC, only VAD mode is supported; Manual mode is not supported. Audio is transmitted via the audio track, without sending input_audio_buffer.append; video is transmitted via the video track, without sending input_image_buffer.append.

Model Event Examples

Audio and video are transmitted via media tracks, without using WebSocket’s media send events.
The following example demonstrates the session.created → session.update → session.updated process for qwen3.8-omni-flash-realtime. Configuration fields should follow this model’s event reference. After receiving session.updated confirming that configuration is complete, mount the upstream media track. See full example for the code. For the example model qwen3.8-omni-flash-realtime, when audio output is enabled, you can use response.audio.done to check the model’s audio generation status; this event does not indicate that the client has finished playing the audio.

Start media transmission and reception

After the model is ready, attach the audio and video tracks needed for upstream transmission. Handle model output in the remote track callback and continue parsing DataChannel events.
  • Audio travels over RTP media tracks; no input_audio_buffer.append event is required.
  • Images travel over video tracks. WebRTC does not support input_image_buffer.append.
  • DataChannels carry text, status, control, and error events.
In the complete example, startSendingMedia() attaches the upstream track, and ontrack processes the received media.

6. End interaction and release resources

When the connection is no longer needed, stop local capture tracks owned by your application, close DataChannels and the RTCPeerConnection, release players, and clear application state. Clear model readiness after a failed or unexpected disconnection. Initialize the model again on a new connection instead of reusing the previous connection's session state. In the complete example, endSession() is responsible for releasing the connection and media resources. Call endSession() when the user ends the call or leaves the page. If your application adds the Canvas or recording features from the best practice, also cancel animation loops and stop Canvas media tracks and the MediaRecorder.

Send and receive events over the Data channel

Check that the channel is open before calling send with a serialized event. Parse incoming messages in its message callback, including events on server-created channels. The names, fields, parameters, and timing of sent events follow the target model’s Client Event Definitions; the parsing method, state changes, results, and error handling after receiving messages adhere to the model’s Server Event Definitions. Initialization events are sent as needed according to the model’s requirements, while subsequent text and control events are sent based on business needs.

DataChannel example

Use the same binding function for client-created and server-created channels. It handles transport and JSON parsing; handleModelEvent in section 5 handles model semantics. Use sendModelEvent for model initialization, text, or control events only when permitted by the model's protocol. In the complete example, sendModelEvent() and bindDataChannel() send and receive events. In the complete example, the start button calls startSession() and the end button calls endSession(). These are functions defined in this example, not built-in WebRTC APIs.

Qwen-Omni-Realtime Event Definitions

Model or applicationClient events (send)Server events (receive)
Qwen-Omni-RealtimeClient EventsServer Events
Also follow these rules:
  • Use the type field of events to identify model events, and handle associated identifiers according to this model’s event reference.
  • Operations such as initialization confirmation, input submission, response cancellation, and task completion—required by the model—are all handled within the Data channel according to the corresponding protocol; these constitute the model’s interaction semantics and are distinct from disconnecting the transport connection.
  • Audio and video data are transmitted via media tracks. When referencing the model event documentation, media encapsulation and available interaction modes still follow the access methods outlined here and do not directly adopt WebSocket’s media transmission approach.

Event references

Scenario Tutorials