Skip to main content

Qwen-Audio-Realtime WebRTC access

Qwen-Audio-Realtime WebRTC access

WebRTC uses media tracks to transmit audio and DataChannel to transmit model events and text. The web client can use the browser’s native API, while other clients can use libraries that support standard WebRTC. Model Studio does not provide a dedicated WebRTC SDK. This document provides the WebRTC integration procedure for this model, along with a complete browser-side example.

Scope of Application

This document is intended for Qwen-Audio-Realtime. Existing integration instructions cover: qwen-audio-3.1-realtime-plus, qwen-audio-3.1-realtime-flash, qwen-audio-3.0-realtime-plus, qwen-audio-3.0-realtime-flash. The example model is qwen-audio-3.1-realtime-plus.
  • Use the API Key and Workspace ID corresponding to your selected region and workspace.
  • Beijing domain: {WorkspaceId}.cn-beijing.maas.aliyuncs.com; Singapore domain: {WorkspaceId}.ap-southeast-1.maas.aliyuncs.com.
  • Long-term API Keys are stored on the server side and should not be included in browser or mobile applications. For instructions on generating temporary API Keys and their permissions, see Generate Temporary API Key.
When capturing audio in the browser, use a secure context; when using the microphone, request microphone permissions. When receiving audio, handle browser autoplay restrictions.

Authentication and SDP Exchange

The browser submits an Offer SDP to the business AppServer. The AppServer forwards the following request using the API Key and returns the Answer SDP unchanged to the browser; long-term API Keys do not reach the frontend.
  • Method: POST
  • Beijing address: https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api/v1/webrtc/realtime?model=qwen-audio-3.1-realtime-plus
  • Singapore address: https://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api/v1/webrtc/realtime?model=qwen-audio-3.1-realtime-plus
  • Request headers: Authorization: Bearer <API_KEY>, Content-Type: application/sdp
  • Request body: Offer SDP string.
  • Successful response: HTTP 200, with the Answer SDP in the body; failure returns 4xx.
WebRTC performs authentication during the SDP exchange and does not require a prior AOQ Token. In the examples below, /api/realtime/sdp refers to a custom interface built by the business AppServer, which must implement the aforementioned forwarding process—it is not a URL provided by Model Studio.

Workflow

WebRTC connection sequence
  1. Prepare the API key and target model parameters on AppServer.
  2. Create the client connection object and register callbacks.
  3. Configure media directions and DataChannels without attaching upstream media tracks.
  4. Exchange SDP through AppServer and wait for the connection and event channel to become ready.
  5. Configure the model according to its protocol. Attach the required upstream media tracks after confirmation.
  6. Stop capture, close the connection, and release resources when interaction is complete.
  • Audio is transmitted via WebRTC media tracks; the application is responsible for obtaining the local audio track, binding the remote player, and sending/receiving DataChannel messages according to the target model’s event definitions.
  • WebRTC authentication takes place during SDP exchange; no AOQ token is required. Proxy authentication and SDP through AppServer to keep the long-term API key out of frontend code.
  • A successful connection indicates that the transmission channel has been established, but it does not mean the model has completed initialization. When using qwen-audio-3.1-realtime-plus on this page, start the required upstream media only after sending session.update and receiving session.updated.

Complete Example

The following is a complete browser-side example. First, implement the /api/realtime/sdp interface of the business AppServer as described in Authentication and SDP Exchange; then open this HTML page from the same origin as the business AppServer, ensuring that the protocol, host, and port match. For example, both the page and /api/realtime/sdp are served by http://localhost:3000. The server is responsible for storing the API Key and forwarding the SDP; do not include long-term API Keys in the page. Click “Start” to establish the connection; click “End” to release resources when the interaction concludes. This example establishes one connection at a time; reload the page to initiate another session. 
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>WebRTC access example</title>
</head>
<body>
<button id="start-session" type="button">Start</button>
<button id="end-session" type="button">End</button>
<script>
const pc = new RTCPeerConnection({ iceServers: [ ] });

const channels = new Set();
const senders = new Map();
const remoteAudio = document.createElement("audio");
remoteAudio.controls = true; // Allow manual playback if autoplay is blocked.
remoteAudio.autoplay = true;
document.body.appendChild(remoteAudio);

let localStream = null;
let eventChannel = null;
let stopped = false;
let started = false;
let sessionCreated = false;
let updateSent = false;
let modelReady = false;
let mediaStarted = false;
const requestController = new AbortController();
const media = {
  sendAudio: true,
  receiveAudio: true,
};

function mediaDirection(send, receive) {
  if (send && receive) return "sendrecv";
  if (send) return "sendonly";
  if (receive) return "recvonly";
  return "inactive";
}

async function configureMedia() {
  // Keep m=audio. The target service must support the selected directions.
  const transceiver = pc.addTransceiver("audio", {
    direction: mediaDirection(media.sendAudio, media.receiveAudio),
  });
  if (media.sendAudio) senders.set("audio", transceiver.sender);
  if (!media.sendAudio) return;
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: true,
    video: false,
  });
  // The user may end the call while the permission prompt is open.
  if (stopped) {
    stream.getTracks().forEach(track => track.stop());
    throw new Error("Session ended");
  }
  localStream = stream;
  // sender.track is still empty, so no captured data is sent to the model.
}
// See Send and receive events over DataChannels for bindDataChannel.
pc.ondatachannel = ({ channel }) => bindDataChannel(channel);
const clientChannel = pc.createDataChannel("oai-events");
bindDataChannel(clientChannel);
pc.onconnectionstatechange = () => {
  console.log("WebRTC state:", pc.connectionState);
  if (pc.connectionState === "connected") {
    tryInitializeModel();
  } else if (["failed", "disconnected", "closed"].includes(pc.connectionState)) {
    // This example ends the session; implement recovery as needed.
    endSession();
  }
};

function waitForIceComplete() {
  return new Promise((resolve, reject) => {
    const timer = setTimeout(() => finish(new Error("ICE gathering timed out")), 15000);
    function finish(error) {
      clearTimeout(timer);
      pc.removeEventListener("icegatheringstatechange", check);
      requestController.signal.removeEventListener("abort", cancel);
      error ? reject(error) : resolve();
    }
    function check() {
      if (pc.iceGatheringState === "complete") finish();
    }
    function cancel() { finish(new Error("Session ended")); }
    pc.addEventListener("icegatheringstatechange", check);
    requestController.signal.addEventListener("abort", cancel, { once: true });
    if (stopped) cancel();
    else check();
  });
}

function normalizeAnswerSdp(sdp) {
  return String(sdp).trim().replace(/\r?\n/g, "\r\n") + "\r\n";
}

async function startSession() {
  if (started || stopped) return;
  started = true;
  try {
    await configureMedia();
    if (stopped) return;
    const offer = await pc.createOffer();
    await pc.setLocalDescription(offer);
    await waitForIceComplete();

    // Implement this same-origin application endpoint; it is not a Model Studio API.
    // AppServer forwards SDP with the API key and returns the raw Answer SDP.
    const response = await fetch("/api/realtime/sdp", {
      method: "POST",
      headers: { "Content-Type": "application/sdp" },
      body: pc.localDescription.sdp,
      signal: requestController.signal,
    });
    if (!response.ok) {
      throw new Error(`SDP exchange failed:${response.status} ${await response.text()}`);
    }
    const answerSdp = normalizeAnswerSdp(await response.text());
    if (stopped) return;
    await pc.setRemoteDescription({ type: "answer", sdp: answerSdp });
    // Continue through connectionstatechange, DataChannel, and model events.
  } catch (error) {
    if (!stopped) console.error("Connection failed:", error);
    endSession();
  }
}
function tryInitializeModel() {
  if (stopped || updateSent || !sessionCreated ||
      pc.connectionState !== "connected" ||
      eventChannel?.readyState !== "open") return;
  sendModelEvent({
    event_id: `event_${crypto.randomUUID()}`,
    type: "session.update",
    session: {
      "modalities": media.receiveAudio ? ["audio", "text"] : ["text"],
      "voice": "longanqian_v3.1",
    },
  });
  updateSent = true;
}

async function handleModelEvent(event, channel) {
  if (stopped) return;
  if (event.type === "session.created") {
    sessionCreated = true;
    eventChannel = channel; // Send configuration through the channel that receives session events.
    tryInitializeModel();
  } else if (event.type === "session.updated" && updateSent) {
    modelReady = true;
    await startSendingMedia();
  } else if (event.type === "error") {
    console.error("Model error:", event.error);
    endSession();
  } else {
    // Display text, transcripts, or response status as defined by the selected model. Receive audio over media tracks.
    console.log("Model event:", event);
  }
}
async function startSendingMedia() {
  if (stopped || !modelReady || mediaStarted) return;
  mediaStarted = true;

  for (const track of localStream?.getTracks() ?? []) {

    if (stopped) return;
    const sender = senders.get(track.kind);
    if (sender) await sender.replaceTrack(track);
  }
}

// Register before startSession(). Tracks without streams are also received correctly.
const remoteAudioStream = new MediaStream();
pc.ontrack = ({ track }) => {
  if (stopped) return;
  if (track.kind === "audio" && media.receiveAudio) {
    remoteAudioStream.addTrack(track);
    remoteAudio.srcObject = remoteAudioStream;
    remoteAudio.play().catch(() => {
      console.info("Autoplay is blocked. Use the audio controls to play.");
    });
  }
};
function endSession() {
  if (stopped) return;
  stopped = true;
  modelReady = false;
  sessionCreated = false;
  updateSent = false;
  mediaStarted = false;
  requestController.abort(); // Cancel pending SDP requests and ICE gathering.
  localStream?.getTracks().forEach(track => track.stop());
  localStream = null;
  channels.forEach(channel => channel.close());
  channels.clear();
  eventChannel = null;
  pc.close();
  senders.clear();
  remoteAudioStream.getTracks().forEach(track => track.stop());
  remoteAudio.pause();
  remoteAudio.srcObject = null;
  remoteAudio.remove();
}
function sendModelEvent(event, channel = eventChannel) {
  if (stopped || pc.connectionState !== "connected" ||
      channel?.readyState !== "open") {
    throw new Error("Data channel is not ready");
  }
  channel.send(JSON.stringify(event));
}

function bindDataChannel(channel) {
  channels.add(channel);
  channel.onopen = () => {
    if (stopped) return;
    console.log("DataChannel opened:", channel.label);
    tryInitializeModel();
  };
  channel.onmessage = ({ data }) => {
    if (stopped) return;
    let event;
    try { event = JSON.parse(data); }
    catch (error) {
      console.warn("Cannot parse model event:", error);
      return;
    }
    if (!event || typeof event !== "object") return;
    handleModelEvent(event, channel).catch(error => {
      if (!stopped) console.error("Model event handling failed:", error);
      endSession();
    });
  };
  channel.onerror = error => console.error("DataChannel error:", error);
  channel.onclose = () => {
    channels.delete(channel);
    if (channel === eventChannel) endSession();
  };
  if (channel.readyState === "open") channel.onopen();
}

document.getElementById("start-session").addEventListener("click", startSession);
document.getElementById("end-session").addEventListener("click", endSession);
</script>
</body>
</html>

1. Prepare authentication and connection parameters

AppServer configures SDP exchange for the target model, region, and workspace. The following parameters apply to standard Realtime connections. For application-specific parameters, follow the corresponding best practice.
SettingValue
MethodPOST
URLhttps://{endpoint}/api/v1/webrtc/realtime?model={model_name}
Content-Typeapplication/sdp
AuthorizationBearer <API_KEY>
Request bodyOffer SDP
Successful responseAnswer SDP
Complete regional, address, and authentication details can be found in Authentication and SDP Exchange. The business AppServer’s interface is implemented by the business itself and is not part of Model Studio’s SDK API.

2. Initialize the Connection Object and Register Callbacks

On the web side, create a RTCPeerConnection; on other platforms, perform the corresponding library initialization. Before establishing the connection, register the following handlers:
  • Connection state changes: Determine whether the connection succeeded, failed, or was disconnected.
  • DataChannel establishment, opening, messages, and closure: carries model events.
  • Remote media track: receives and plays audio, or handles other media supported by the target application.
The complete example creates the connection object, state variables, and callbacks within the same script. Each call creates a single connection; after ending, reinitialize these objects to make another call. The connection object and state variables are detailed in Complete Example. Message listeners should be registered before model initialization to avoid missing session events proactively sent by the server. Do not listen only to channels created by the client: the server will push events through a channel named txt, so you must handle the datachannel callback.

3. Configure media directions and DataChannels

Selecting Audio Direction

Directions are relative to the client. WebRTC transceivers negotiate sending and receiving: sendrecv is bidirectional, sendonly sends only, and recvonly receives only. Available capabilities also depend on the server's SDP answer and model support.
DirectionConfiguration
Send audio negotiates audio transmission capabilities and prepares the microphone or external audio track; attach the track after the model is ready.
pulls audio— negotiates audio reception capabilities and binds audio playback in the remote track callback
For example, voice conversations typically use bidirectional audio. The Offer SDP must include the audio media section m=audio; do not remove this section simply because the business does not need to play audio.

Media Direction for This Model

Model or applicationSend audioReceive audioSend videoReceive video
Qwen-Audio-RealtimeEnable for voice inputEnable for voice outputNot supportedNot supported
This table describes the media directions needed by the application; it does not guarantee that every combination can be negotiated independently. Configuration depends on the target model and server negotiation capabilities.

Prepare media without sending it

Negotiate the required directions before creating the Offer. For models that require initialization confirmation, reserve sending capabilities without attaching captured tracks. After the model is ready, use replaceTrack on the corresponding sender to start transmission. Register media reception handlers in advance. Audio transmission and reception should be configured separately as needed; do not combine microphone activation, audio playback, and model initialization into a single operation.

Media configuration example

The following example illustrates bidirectional voice. Two switches in media control audio transmission and reception respectively; addTransceiver pre-negotiates the direction, and the captured track is first stored locally until the model is ready, at which point it is attached to the sender. See full example for the code.

Create event channels

Create a DataChannel before creating an Offer so that the SDP includes data channel negotiation. Refer to Data Channel Send and Receive Events for how the client and server handle channel creation. The DataChannel is responsible for sending model initialization, text, and control events, and for receiving model results, status, and errors. The media track is responsible for audio transmission. The complete example creates a client event channel and registers the server-side channel callback before creating an Offer.

4. Establish Connection

Complete the connection in the following order:
  1. Call createOffer and setLocalDescription.
  2. Collect candidate information according to the ICE methods supported by the server; in this example, the local SDP is sent after ICE collection is complete.
  3. Send the local SDP to the business AppServer, which uses the API Key to request the Model Studio SDP exchange interface.
  4. Check the HTTP status, and upon success, call setRemoteDescription using the returned Answer SDP.
  5. Wait for the connection status to become successful, and confirm that the DataChannel used for sending model events has been opened.
SDP exchange success or completion of setRemoteDescription only indicates that the corresponding negotiation step has been completed; it cannot be directly taken as proof that the media connection has succeeded. Model events also cannot be sent when the DataChannel is not in the open state.

Connection Code and Status Callbacks

Listen to connectionstatechange to determine the connection status, and listen to icegatheringstatechange to wait for the complete local SDP. Model initialization also requires the open of the DataChannel and the model’s ready event; it should not rely solely on the completion of SDP exchange for judgment. The full example completes SDP exchange via startSession() and handles the connection result through the connection status callback. The business AppServer should forward the Offer according to the address and request headers specified in Section 1; the sample API is expected to return plain SDP text. Do not pass content containing HTTP response headers, logs, or JSON wrappers directly to setRemoteDescription.

5. Exchange model events and media

Initialize the model

When the connection and event channel are available, initialize the target model according to its protocol. Store server initialization events and combine them with channel state to advance the flow; do not assume a fixed callback arrival order. For example, when configuring a model using session.update, construct the event according to its protocol and wait for configuration confirmation before sending input. Parameters such as voice, output modalities, VAD, and sampling rate should all be derived from the current model’s documentation. session.created indicates session creation; it should not be uniformly regarded as configuration having taken effect. For models that require confirmation of session configuration, wait for session.updated. The definitions of model events are provided in Client Events and Server Events. Media transmission and available modes should be handled as described on this page.

Model Event Examples

Audio is transmitted via the media track and does not use WebSocket’s media send event.
The following example demonstrates the session.created → session.update → session.updated flow for qwen-audio-3.1-realtime-plus. Configure fields according to the model’s event reference. Only after receiving session.updated confirming that the configuration is complete should you attach the upstream audio track. See full example for the code. For the example model qwen-audio-3.1-realtime-plus, enabling audio output allows you to monitor the model’s audio generation status via response.audio.done; this event does not indicate that the client’s audio has finished playing.

Start media transmission and reception

Once the model is ready, attach the required upstream audio track. Handle the model’s output via the remote track callback and continuously parse the events returned by the DataChannel.
  • Audio travels over RTP media tracks; no input_audio_buffer.append event is required.
  • DataChannels carry text, status, control, and error events.
In the complete example, startSendingMedia() attaches the upstream track, and ontrack processes the received media.

6. End interaction and release resources

When the connection is no longer needed, stop local capture tracks owned by your application, close DataChannels and the RTCPeerConnection, release players, and clear application state. Clear model readiness after a failed or unexpected disconnection. Initialize the model again on a new connection instead of reusing the previous connection's session state. In the complete example, endSession() is responsible for releasing the connection and media resources. Call endSession() when the user clicks the end button or the page is closed.

Send and receive events over the Data channel

Check that the channel is open before calling send with a serialized event. Parse incoming messages in its message callback, including events on server-created channels. The names, fields, parameters, and timing of sent events follow the target model’s Client Event Definitions; the parsing method, state changes, results, and error handling after receiving messages adhere to the model’s Server Event Definitions. Initialization events are sent as needed according to the model’s requirements, while subsequent text and control events are sent based on business needs.

DataChannel example

Use the same binding function for client-created and server-created channels. It handles transport and JSON parsing; handleModelEvent in section 5 handles model semantics. Use sendModelEvent for model initialization, text, or control events only when permitted by the model's protocol. In the complete example, sendModelEvent() and bindDataChannel() send and receive events. In the complete example, the start button calls startSession() and the end button calls endSession(). These are functions defined in this example, not built-in WebRTC APIs.

Qwen-Audio-Realtime event definitions

Model or applicationClient events (send)Server events (receive)
Qwen-Audio-RealtimeClient eventsServer events
Also follow these rules:
  • Use the type field of events to identify model events, and handle associated identifiers according to this model’s event reference.
  • Operations such as initialization confirmation, input submission, response cancellation, and task completion—required by the model—are all handled within the Data channel according to the corresponding protocol; these constitute the model’s interaction semantics and are distinct from disconnecting the transport connection.
  • Audio data is transmitted over media tracks. When using the model event reference, follow this page for media encapsulation and available interaction modes. Do not directly apply the WebSocket method for sending media.

Event references