Skip to main content

Qwen-Omni-Realtime AOQ access

Qwen-Omni-Realtime AOQ access

AOQ transmits audio/video media and model events over separate tracks. SDK handles media capture, encoding/decoding, transmission, and playback; the application sends model events via the Data track and processes responses. This page uses the media direction and event protocol of Qwen-Omni-Realtime.

Scope of Application

This document is aimed at Qwen-Omni-Realtime. Existing integration documentation covers: qwen3.8-omni-flash-realtime, qwen3.5-omni-plus-realtime, qwen3.5-omni-flash-realtime. The example model is qwen3.8-omni-flash-realtime.
  • Use the API Key and Workspace ID corresponding to your selected region and workspace.
  • Beijing domain: {WorkspaceId}.cn-beijing.maas.aliyuncs.com; Singapore domain: {WorkspaceId}.ap-southeast-1.maas.aliyuncs.com.
  • Store the API Key only on the server. Do not embed it in a browser or mobile application.
  • When capturing audio with a microphone or video with a camera, request the necessary permissions according to platform requirements.
  • Download the SDK for your platform from SDK downloads.

Authentication and connection credentials

Send the following request from your AppServer. First configure the API Key as an environment variable. Replace {WorkspaceId} in the domain with your workspace ID. For another region, use the corresponding domain listed above. clientIp is optional. If omitted, the gateway uses the IP address from which it receives the request.
export ENDPOINT="{WorkspaceId}.cn-beijing.maas.aliyuncs.com"
curl -X POST \
  "https://${ENDPOINT}/api/v1/webrtc/realtime?model=qwen3.8-omni-flash-realtime" \
  -H "Authorization: Bearer ${DASHSCOPE_API_KEY}" \
  -H "Content-Type: application/json" \
  -H "x-dashscope-rtc-transport: moq" \
  -d '{}'
To specify the client public IP address, replace -d '{}' in the request body with -d '{"clientIp":"<CLIENT_PUBLIC_IP>"}'.

Request fields

Configuration

Value

Description

endpoint

Select the endpoint domain for your use case

Specify the endpoint domain. For details, see Select a region, deployment scope, and endpoint. If you use Token Plan, set WorkspaceId in the endpoint domain to token-plan.

Content-Type

application/json

Message media type

Authorization

Bearer <YOUR_API_KEY>

Your API Key

x-dashscope-rtc-transport

moq

Select the AOQ protocol

clientIp

The client public IP address

Optional. If omitted, the service uses the IP address from which the gateway receives the request. If specified, it uses clientIp. The service selects the best Relay endpoint based on the client IP address.

Response example

{
    "sid": "1d06b55683db49bba67a407902f62d02:1782706970:69aecdc5...",
    "aoqTokenForClient": "ecc1a46015d5496ca4ff7a48281eb739",
    "clientRelayEndpoints": [{"endpoint": "121.199.XX.XX", "port": 8443, "route_index": 0}],
    "clientRelayCertFingerprint": "sha256/99843495...",
    "sidExpiresInSecs": 7200,
    "extraInfo": {"workspaceIdHash": "2021b6f98cea4cff"}
}

Response fields

Field

Description

sid

Unique session identifier.

aoqTokenForClient

Client connection token. Pass this value to the SDK token field.

clientRelayEndpoints

Array of Relay endpoints. Each element in the response example contains the following fields:
endpointRelay endpoint address.portRelay endpoint port.route_indexThe iOS example below uses the array index when this field is absent.

clientRelayCertFingerprint

Relay TLS certificate fingerprint.

sidExpiresInSecs

Session expiration time in seconds.

extraInfo

Additional information.
workspaceIdHashWorkspace ID hash.

AOQ Client SDK connection example

clientIp is an optional field in the request body. If not specified, the IP address that requests the Model Studio gateway is used as the client IP. If specified, the clientIp value takes precedence. Have your business AppServer obtain the client's real IP address and pass it in to get the best Relay access point.
iOS (Swift)
let resp = try JSONDecoder().decode(AllocateResponse.self, from: responseData)

let config = AoqConnectConfig()
config.token = resp.aoqTokenForClient
config.sid = resp.sid
config.certFingerprint = resp.clientRelayCertFingerprint
config.relayEndpoints = resp.clientRelayEndpoints.enumerated().map { index, item in
    let ep = AoqRelayEndpoint()
    // Fall back to the array index when route_index is missing
    ep.routeIndex = item.routeIndex ?? index
    ep.endpoint = item.endpoint
    ep.port = item.port
    return ep
}
config.workspaceIdHash = resp.extraInfo?.workspaceIdHash ?? ""

let audioTrack = AoqTrackParam()
audioTrack.trackType = .audio
let dataTrack = AoqTrackParam()
dataTrack.trackType = .data
config.publishTracks = [audioTrack, dataTrack]
config.subscribeTracks = [audioTrack, dataTrack]

engine.connect(config)

Workflow

AOQ connection sequence
  1. Obtain connection credentials through AppServer.
  2. Create the SDK engine and register callbacks.
  3. Configure media and Data tracks, and disable upstream media transmission.
  4. Establish the AOQ connection and wait for it to connect.
  5. Configure the model according to its protocol. Enable the required upstream media after confirmation.
  6. Disconnect and release resources when interaction is complete.
  • AOQ SDK handles media capture, processing, transmission, reception, and playback. Your application must send and receive Data messages according to the target model's event definitions.
  • A successful connection indicates that the transmission channel to the model has been established; the client should enable the required upstream media only after the model initialization is complete. For the example model qwen3.8-omni-flash-realtime on this page, wait for session.updated after sending session.update, then begin media transmission.

1. Obtain connection credentials

The client requests credentials from your AppServer. The AppServer calls this model's Realtime connection endpoint and includes the API Key and x-dashscope-rtc-transport: moq in the gateway request. For the full endpoint, request, and field descriptions, see Token authentication. Map the response to AoqConnectConfig:
Gateway responseSDK configuration
aoqTokenForClienttoken
sidsid
clientRelayCertFingerprintcertFingerprint
clientRelayEndpointsrelayEndpoints, convert each access point according to the structure
extraInfo contains workspaceIdHashworkspaceIdHash
Obtain new credentials for each new connection. Whether a session can be reused or another task can start within the same connection depends on the target model's protocol.

2. Initialize the SDK and register callbacks

Create the engine and register callbacks for connection status, Data messages, and errors. The following example uses Swift on iOS:
let createConfig = AoqCreateConfig()
createConfig.workDir = workDir
createConfig.enableDumpAudio = false
engine = AoqClientEngine.createEngine(createConfig, delegate: self)
Implement AoqEngineDelegate; maintain connection status in onConnectionStatusChange; parse target model events in onDataMsg; handle SDK errors in onError. Error events returned by the model also need to be handled separately within the Data message processing logic. For details on the SDK interface and error descriptions, see Platforms SDK and Feature Reference.

3. Configure the SDK

Select media directions

Directions are relative to the client. Configure only the input and output modalities that your application needs. Do not add tracks for unused directions.
DirectionConfigurationTypical use
pushes audiopublishes the Audio track, sets the audio encoder, configures built-in or external capturevoice conversation
pulls audiosubscribes to the Audio track, sets the audio decoder, configures the player or external playbackvoice response
pushes videopublishes the Video track, configures the camera or external video input and encoding parameterssends video to models that support visual input
pulls videothis model does not support video outputdo not configure video subscription
Data tracks are independent of these four directions: publish a Data track to send model events and subscribe to receive responses. Text input and output also require Data tracks.

Media directions for this model

The table below lists the media directions for this model. See Applicable Scope for the model’s range; for other integration methods, refer to Qwen-Omni-Realtime.
Model or applicationSend audioReceive audioSend videoReceive video
Qwen-Omni-Realtimeenable when voice input is requiredenable when voice response is neededenable when visual input is requiredcannot enable

Configure media parameters and tracks

Configure upstream and downstream audio separately: use setAudioEncoderConfig for upstream audio and setAudioDecoderConfig for downstream audio. These parameters control network transmission; the gateway converts audio to the format required by the model. Configure transport encoding and the audio format in model events according to the SDK and target model requirements, respectively. Start startAudioCapture, startAudioPlayer, or startVideoCapture as needed. For custom input and output, see Advanced features below. publishTracks are sent by the client, and subscribeTracks are received by the client:
ScenariopublishTrackssubscribeTracks
voice conversationAudio, DataAudio, Data
voice conversation with visual inputAudio, Video, DataAudio, Data
Disable upstream media before connecting to prevent input from reaching the model before it is ready:
engine.enableSendMediaStream(.audio, enable: false)
engine.enableSendMediaStream(.video, enable: false)
Capture, track publication, and permission to send are separate operations. Disabling transmission does not stop local capture or affect Data events. Enabling transmission does not create tracks that were not added.

4. Establish Connection

Fill the credentials and track configuration into AoqConnectConfig, then call:
engine.connect(connectConfig)
Wait for onConnectionStatusChange to return .connected, then proceed to the model initialization process. A return from connect does not mean that the connection has succeeded.

5. Exchange model events and media

Send and parse model events

The client sends events via the Data track. The following is a Swift code snippet for iOS; for other interfaces, see iOS SDK Reference:
// Build eventJSON according to the target model client event protocol.
let msg = AoqDataMsg()
msg.data = eventJSON.data(using: .utf8)!
engine.send(msg)
Parse the response in onDataMsg. This model identifies event types through the type field. Complete fields, parameters, and termination conditions should follow the target model’s documentation.

Enable media after the model is ready

After the current model, session, or task satisfies its readiness conditions, enable only the configured upstream directions. For speech input, call:
engine.enableSendMediaStream(.audio, enable: true)
Enable .video only when video input is required. Transmit audio and video on their media tracks, and model events and text on Data tracks. For example, Omni over AOQ does not require input_audio_buffer.append or input_image_buffer.append to send the same media again. Handle downstream audio with the configured player or external playback logic.

6. End interaction and release resources

When the connection is no longer needed, call the AOQ SDK disconnect and resource-release interfaces from your application:
engine.disconnect()
AoqClientEngine.destroy()
After an unexpected disconnection, clear the application's readiness state and follow Connection state management. Initialize the model again after reconnecting.

Sending and Receiving Events via the Data Channel

The client sends and receives model events via the Data track of AOQ SDK. During connection, both publishing and subscribing to the Data track are required; when sending, serialize the model event into AoqDataMsg and call the platform’s Data send interface; upon receipt, parse the message in the onDataMsg callback. See Platform-Specific SDK for detailed interfaces. The names, fields, parameters, and timing of sent events follow the target model’s Client Event Definitions; the parsing method, state changes, results, and error handling after receiving messages adhere to the model’s Server Event Definitions. Initialization events are sent as needed according to the model’s requirements, while subsequent text and control events are sent based on business needs.
Model or applicationClient events (send)Server events (receive)
Qwen-Omni-RealtimeClient EventsServer Events
Also follow these rules:
  • Use the type field of events to identify model events, and handle associated identifiers according to this model’s event reference.
  • Operations such as initialization confirmation, input submission, response cancellation, and task completion—required by the model—are all handled within the Data channel according to the corresponding protocol; these constitute the model’s interaction semantics and are distinct from disconnecting the transport connection.
  • Before ending, complete input submission and result reception as described in this model’s event documentation; if full playback is required, release the connection only after processing any locally queued audio.
  • Audio and video data are transmitted via media tracks. When referencing the model event documentation, media encapsulation and available interaction modes still follow the access methods outlined here and do not directly adopt WebSocket’s media transmission approach.

Advanced features

CapabilityUse caseDocumentation
Control audio and video transmission separatelyWait for model readiness, pause upstream media, or enable video as neededMedia transmission management
Audio devices and processingEncoding, decoding, speakers, file mixing, and audio frame callbacksCommon audio features
External audio inputUse an application-provided audio sourceCustom audio capture
External audio outputProcess or play output audio in your applicationCustom audio playback
Video capture and inputUse a camera or application-provided imagesCommon video features, Custom video input
Media transmission controls affect media only. Model response cancellation and turn submission still require the target model's events.

Session Configuration Example

The following example applies to qwen3.8-omni-flash-realtime. Audio and video are transmitted via media tracks; the media send events of WebSocket are not used.
{
  "type": "session.update",
  "session": {
    "modalities": [
      "text",
      "audio"
    ],
    "voice": "Tina",
    "turn_detection": {
      "type": "server_vad"
    }
  }
}

AOQ SDK downloads

The version update notes below describe the general capabilities of AOQ SDK; the scope of media support for this model is detailed in the Media Orientation for This Model above.
VersionPlatformDownloadRelease DateRelease Notes
v1.4.0AndroidAoqClientSdk-release.aarlibPluginOpus.zip2026-10-27
  1. Added support for externally supplied H.264 video.
  2. Improved performance.
  3. Fixed known stability issues.
iOSAoqClientSdk.framework.zipAoqScreenShare.framework.zipPluginOpus.framework.zip
HarmonyOSAoqClientSdk.harlibPluginOpus.zip
WindowsAoqClientSdk-windows-x64.zip
macOSAoqClientSdk.framework.zipPluginOpus.framework.zip
Electronaoq-electron-sdk-1.4.0-59463182.tgzAlternatively, install through npm:npm install aoq-electron-sdk@1.4.0-59463182
Linuxx86_64: AoqClientSdk.ziparm64: AoqClientSdk.zip
v1.3.0Android2026-09-18
  1. Added screen sharing support.

  2. Deprecated bitcode.

  3. Added support for remotely loading .so libraries.

  4. Added support for externally setting the maximum value for encoded frames.

  5. Changed the default codec to Opus.

  6. Improved performance.

  7. Fixed known stability issues.

iOS
HarmonyOS
WindowsAoqClientSdk-windows-x64.zip
macOS
Electron
Or install with npm: npm install aoq-electron-sdk@1.3.0-58586345
Linux
v1.2.2iOSAoqClientSdk.framework.zip2026-09-09
  1. Fixed an issue where an iOS call could not resume after an interruption by the system or a third-party voice call ended.

  2. Improved echo cancellation at the start of iOS calls.

v1.2.0Android 2026-08-17
  1. Added support for macOS

  2. Added support for Windows

  3. Added support for Electron

  4. Added Python and C++ APIs for Linux

  5. Added volume level detection

  6. Improved performance

  7. Fixed known stability issues

iOS
HarmonyOS
Windows
macOS
Electron
Alternatively, install it by using npm: npm install aoq-electron-sdk@1.2.0-57407598
Linux
v1.1.0Android 2026-07-31
  1. Supports ASR models

  2. Supports TTS models

  3. Supports the multimodal interaction toolkit

  4. Added support for the Linux platform

  5. Improved performance

  6. Fixed known stability issues

iOS
HarmonyOS
LinuxAoqClientSdk.zip
v1.0.1Android 2026-07-09
  1. Access to models and applications via AOQ protocol

  2. Audio and video codec parameters

  3. Audio and video device capture and playback

  4. Custom external audio capture

  5. Custom external video capture

  6. Custom external video encoding

  7. Media stream sending control

iOS
HarmonyOS
AOQ loads the Opus codec as an audio plug-in. To use Opus encoding or decoding, download the Opus plugin and add it to your project.

Event references

Scenario Tutorials