Qwen-Audio-Realtime WebSocket access
qwen-audio-3.1-realtime-plus, qwen-audio-3.1-realtime-flash, qwen-audio-3.0-realtime-plus, and qwen-audio-3.0-realtime-flash.
Endpoint
Use the following WebSocket URL. The model query parameter specifies the model. Replace <model_name> with the model name:
- China (Beijing)
- Singapore
wss://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api-ws/v1/realtime?model=<model_name>Replace {WorkspaceId} with your Workspace ID when calling the API.Request headers
Include the following information in the request headers:
Parameter | Type | Required | Description |
|---|---|---|---|
Authorization | string | Yes | Authentication token in the format |
user-agent | string | No | Client identifier used to trace the request source. |
X-DashScope-WorkSpace | string | No | Alibaba Cloud Model Studio workspace ID. |
X-DashScope-DataInspection header for data inspection. Use it according to the model requirements. The example for qwen-audio-3.1-realtime-plus uses x-dashscope-dataInspection: disable.
Core concepts
- Session: A WebSocket connection corresponds to one session, which maintains the configuration and conversation context.
- Conversation Item: Each message in the conversation, organized as a linked list.
- Response: The output of one model inference. A response contains one or more output items, which can be assistant messages or function calls.
- Function Call: A request from the model for the client to execute a tool function. After executing the function, the client submits the result as
function_call_output, then sendsresponse.createto trigger the next inference. - Turn Detection: Controls when inference is triggered.
Interaction modes
The Qwen-Audio Realtime API supports three interaction modes, configured with session.turn_detection in the session.update event. For automatic detection, set type within the object. For push-to-talk, set the entire turn_detection field to null:
Mode | session.turn_detection value | Description | Use cases |
|---|---|---|---|
server_vad | Object. Set | The server uses VAD to detect the start and end of speech and automatically trigger inference. | Hands-free conversation and voice assistants |
Smart semantic turn detection | Object. Set | Combines acoustic perception and semantic understanding to detect turn boundaries, rather than relying only on voice signals. Sounds without semantic meaning, such as "uh" and "ah", do not trigger a turn or interrupt model playback. | Low-latency natural conversation and accurate interruption handling |
push-to-talk |
| The client manually commits audio and triggers inference. | Push-to-talk and precise control |
Interaction flow
For detailed event definitions, see Client events and Server events.
server_vad mode
The server performs voice activity detection on incoming audio and automatically triggers inference when it detects the end of speech.
Enable this mode: Set turn_detection.type to server_vad in the session.update event.
- A complete conversation turn
- User interruption
- The client establishes a WebSocket connection. The server returns
session.created. - The client sends
session.updateto configure the session. The server returnssession.updated. - The client continuously sends
input_audio_buffer.appendto append audio data. - When the server detects the start of speech, it returns
input_audio_buffer.speech_startedand streams ASR transcription deltas inconversation.item.input_audio_transcription.delta. - When the server detects the end of speech, it returns
input_audio_buffer.speech_stopped,input_audio_buffer.committed, andconversation.item.created. - The server automatically generates a response and streams text and audio deltas (
response.audio_transcript.deltaandresponse.audio.delta), then returnsresponse.done.
smart_turn / smart_turn_v2 mode
This mode combines acoustic perception and semantic understanding to detect the end of speech. It can filter meaningless sounds, such as acknowledgments and background audio. Sounds without semantic meaning are passed through in conversation.item.ambient_audio_transcription.delta events and do not trigger a conversation turn.
Enable this mode: Set turn_detection.type to smart_turn or smart_turn_v2 in the session.update event.
smart_turn_v2 (recommended): Builds on smart_turn with stronger understanding of conversation context to better identify interaction intent in multi-party scenarios. It reduces unwanted responses and interruptions caused by other speakers or unrelated remarks.
- A complete conversation turn
- User interruption
- Invalid turn
- Sounds without semantic meaning, such as "uh" and "ah", do not trigger inference. They are returned through
ambient_audio_transcriptionevents. - Speech previously identified as valid may be invalidated:
input_audio_buffer.speech_stoppedreturnsreason=turn_invalid. In this case, inference is not triggered. - While waiting for the user's next input, the client can explicitly send
response.createto trigger inference.
Speaker enhancement configuration flow
In smart_turn / smart_turn_v2 mode, if the first session.update includes voiceprint_audio_urls, the server asynchronously registers the voiceprints by loading the target speakers' audio features. Events report registration progress. A registration failure does not block normal conversation.
Voiceprint registration proceeds in the following order:
-
The client establishes a WebSocket connection. The server returns
session.created. -
The client sends its first
session.update, providing the voiceprint audio URLs inturn_detection.voiceprint_audio_urls. -
The server immediately starts asynchronous voiceprint registration. Before returning
session.updated, it sendsvoiceprint_audio_list.in_progresswith the unique registration task identifieritem_id. -
The server returns
session.updatedto confirm that the session configuration is in effect. -
After voiceprint registration finishes, the server sends a terminal event. Its
item_idmatches the value from step 3:- Registration succeeds:
voiceprint_audio_list.completed. - Registration fails:
voiceprint_audio_list.failed, with areasonfield explaining the failure, such as an audio URL that cannot be downloaded.
- Registration succeeds:
voiceprint_audio_urls takes effect only in the first session.update. The field is ignored in subsequent updates.push-to-talk mode
The client manually controls when to commit audio and trigger inference. This mode is suitable for push-to-talk interactions.
Enable this mode: Set turn_detection to null in the session.update event.
- A complete conversation turn
- User interruption
- The client continuously sends
input_audio_buffer.appendto append audio data. - After the user finishes speaking, the client sends
input_audio_buffer.committo commit the buffer. - The client sends
response.createto trigger inference manually. - The server generates a response and streams text and audio.
Operation constraints by mode
Operation | push-to-talk | server_vad | smart_turn / smart_turn_v2 |
|---|---|---|---|
session.update | All fields can be changed when IDLE; some are restricted otherwise | All fields can be changed when IDLE; some are restricted otherwise | All fields can be changed when IDLE; some are restricted otherwise |
input_audio_buffer.append | Allowed | Allowed | Allowed |
input_audio_buffer.commit | Allowed | Ignored | Ignored |
input_audio_buffer.clear | Allowed | Ignored | Ignored |
response.create | Allowed after committing buffered audio with | Allowed when no response is being generated; do not trigger another response while generation is in progress | Allowed while waiting for the next user input; do not trigger another response within an active turn, from |
response.cancel | Allowed during inference | Allowed during inference | Allowed during inference |
conversation.item.create/delete/retrieve | Allowed | Allowed | Allowed |
turn_detection and input_audio_format can be changed only before audio is first sent, while the session is IDLE.Error handling
If the handshake fails, verify the domain, region, workspace, API Key, model name, model activation status, and permissions. For full error descriptions, see Error codes.
Type | Behavior | Example |
|---|---|---|
Client error ( | The connection remains open; the error is reported | Invalid parameters, operations not allowed in the current state, or duplicate item_id |
Server error ( | The connection is terminated | LLM connection failure or storage failure |