Qwen-Audio client events - Alibaba Cloud Model Studio

User guide: Realtime Audio Chat (Qwen-Audio-Realtime). For event interaction sequences, see WebSocket API.

session.update

Description: After a connection is established, send this event to update the default session configuration. Include only the fields you want to change; omitted fields retain their current values. If any parameter is invalid, the server returns an error. If all parameters are valid, the server applies the changes and returns the full configuration.

turn_detection can only be modified before the first audio is sent (IDLE state).

typestring(required)Event type. Fixed value: session.update.sessionobject (optional)Session configuration.

Properties

modalitiesarray (optional)Output modalities for the model. Valid values:

["text"] Text only.
["audio", "text"] (default) Both text and audio.

voicestring (optional)TTS voice name. Default: longanqian. Two types are supported. Can only be set in the first session.update; ignored in subsequent calls.

System voices: Available values: longanqian, longanlingxin, longanlingxi, longanxiaoxin, longanlufeng.
Cloned voices: created via the Voice Cloning API. Pass the returned voice_id as this parameter's value. For details, see Voice configuration.

enable_speech_emotionboolean (optional)Specifies whether to enable speech emotion enhancement. When enabled, the response voice exhibits more pronounced emotional variations. Default: true. Valid values: true and false.instructionsstring (optional)System instructions that define the model's role, response style, and behavioral preferences. Applies to the entire session.input_audio_formatstring (optional)Input audio format. Currently only pcm (16 kHz, 16-bit, mono) is supported and is the default. Can only be modified before the first audio is sent (IDLE state).output_audio_formatstring (optional)Output audio format. Currently only pcm (24 kHz, 16-bit, mono) is supported and is the default.max_history_turnsinteger (optional)Maximum number of conversation turns (question-answer pairs) included in a single request. Valid values: 1 to 50. Default: 20.enable_searchboolean (optional)Applies only to the qwen-audio-3.0-realtime-plus and qwen-audio-3.0-realtime-flash models.Specifies whether to enable web search. Default: false. When enabled, the model decides whether to search the web based on the real-time query.

tools and enable_search cannot be enabled at the same time.

search_optionsobject (optional)The web search configuration. This parameter takes effect only when enable_search is set to true.

Properties

enable_sourceboolean (optional)Specifies whether to return the sources of search results. Set this parameter to true to return the sources.

toolsarray (optional)Tool definitions for Function Calling. Once configured, the model decides whether to call a tool based on user input.

Properties

typestring(required)Fixed value: function.function.namestring(required)Name of the tool function.function.descriptionstring (optional)Description of the tool function. The model uses this to determine whether to call the tool.function.parametersobject (optional)Description of the tool function's input parameters. The model uses this to extract the required parameters. Omit this field if the function takes no parameters.

Properties

typestring(required)Fixed value: object.propertiesobject (optional)Describes each parameter's name, data type, and description.requiredarray (optional)Specifies which parameters are required.

turn_detectionobject|null (optional)Turn detection configuration. To switch to push-to-talk mode, set this field to null. In push-to-talk mode, audio must be committed manually and inference triggered manually. If this field isn't provided, VAD is enabled with its default parameters.

Properties

typestring (optional)VAD type. Valid values:

server_vad (default): Detects speech onset and offset based on acoustic features and automatically triggers inference.
smart_turn: An intelligent turn detection mode that combines acoustic perception and semantic understanding to determine turn boundaries. Filler sounds such as "um" or "uh" don't trigger a new turn or interrupt model playback.

thresholdfloat (optional)VAD sensitivity. Only effective in server_vad mode (ignored in smart_turn mode). Lower values increase VAD sensitivity, making it easier to detect faint sounds (including background noise) as speech. Higher values decrease sensitivity, requiring clearer and louder speech to trigger detection.Range: [-1.0, 1.0]. Default: 0.5.silence_duration_msinteger (optional)Minimum silence duration (in milliseconds) after speech ends before triggering a model response. Only effective in server_vad mode (ignored in smart_turn mode). Lower values produce faster responses but may cause false triggers during brief pauses.Range: [200, 6000]. Default: 800. Recommended range for conversations: 400-800.voiceprint_audio_urlsarray (optional)Only effective in smart_turn mode. A list of publicly accessible URLs pointing to pre-recorded audio samples from the target user, used for speaker enhancement. Once registered, the model locks onto the target speaker during duplex conversations, effectively ignoring other voices and background noise. Maximum 5 URLs. Audio format requirements: 16kHz PCM or WAV.

This parameter can only be configured in the first session.update event. It is ignored in subsequent session.update events.

{
    "type": "session.update",
    "session": {
        "modalities": [
            "text",
            "audio"
        ],
        "voice": "longanqian",
        "turn_detection": {
            "type": "server_vad",
            "threshold": 0.5,
            "silence_duration_ms": 800
        }
    }
}

Function Calling:

{
    "type": "session.update",
    "session": {
        "modalities": [
            "text",
            "audio"
        ],
        "voice": "longanqian",
        "tools": [
            {
                "type": "function",
                "function": {
                    "name": "get_weather",
                    "description": "Get weather for a specified city",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "city": {
                                "type": "string",
                                "title": "City"
                            }
                        },
                        "required": ["city"]
                    }
                }
            }
        ],
        "turn_detection": {
            "type": "server_vad",
            "threshold": 0.5,
            "silence_duration_ms": 800
        }
    }
}

Voiceprint registration (voiceprint_audio_urls):

{
    "type": "session.update",
    "session": {
        "turn_detection": {
            "type": "smart_turn",
            "voiceprint_audio_urls": [
                "https://example.com/speaker1.pcm",
                "https://example.com/speaker2.wav"
            ]
        }
    }
}

input_audio_buffer.append

Description: Appends audio data to the input buffer. Send this event continuously at a high frequency — for example, one chunk every 20–40 ms. The server sends no acknowledgment for this event.

typestring(required)Event type. Fixed value: input_audio_buffer.append.audiostring(required)Base64-encoded audio data.

{
    "type": "input_audio_buffer.append",
    "audio": "<Base64-encoded audio data>"
}

input_audio_buffer.commit

Description: Push-to-talk mode only. Commits the buffered audio as a user message. This doesn't automatically trigger inference. Send response.create to trigger inference manually. This event is ignored in server_vad and smart_turn modes.

typestring(required)Event type. Fixed value: input_audio_buffer.commit.

{
    "type": "input_audio_buffer.commit"
}

input_audio_buffer.clear

Description: Push-to-talk mode only. Clears uncommitted audio from the buffer. This event is ignored in server_vad and smart_turn modes. The server responds with an input_audio_buffer.cleared event.

typestring(required)Event type. Fixed value: input_audio_buffer.clear.

{
    "type": "input_audio_buffer.clear"
}

conversation.item.create

Description: Inserts a conversation item into the conversation context. Use this event to inject historical context or add text content, or to return Function Calling results.

If item.id already exists in the conversation, an error is returned and the item isn't created.

typestring(required)Event type. Fixed value: conversation.item.create.previous_item_idstring (optional)Specifies the conversation item after which the new item is inserted. If not provided, the item is appended to the end of the conversation.itemobject(required)The conversation item to create.

Properties

idstring (optional)Unique identifier for the conversation item. If not provided, the server generates one automatically. An error is returned if the specified ID already exists in the conversation.typestring(required)Conversation item type. Valid values:

message: A regular conversation message.
function_call: A function call request. Typically generated by the server, but clients can also use it to inject historical context.
function_call_output: A tool execution result. After receiving a function_call, the client executes the tool and returns the result using this type.

rolestring (required for message type)Message role. Valid values: system, user, assistant.contentarray (required for message type)List of message content elements. Each element contains a type field and corresponding data fields.

Supported content types by role

systeminput_text: System message. Required field: text.user

input_text: User text input. Required field: text.
input_audio: User audio input. Required field: audio (Base64-encoded).

assistantoutput_text: Assistant text output. Required field: text.

call_idstring (required for function_call / function_call_output type)Unique identifier for the function call, used to correlate requests and results.namestring (required for function_call type)Name of the function to call.argumentsstring (required for function_call type)Function call parameters in JSON string format.outputstring (required for function_call_output type)Tool execution result in JSON string format.

Inject a user text message:

{
    "type": "conversation.item.create",
    "previous_item_id": "item_xxx",
    "item": {
        "id": "my_item_001",
        "type": "message",
        "role": "user",
        "content": [
            {
                "type": "input_text",
                "text": "Please summarize our last conversation"
            }
        ]
    }
}

Return a Function Calling result:

{
    "type": "conversation.item.create",
    "item": {
        "type": "function_call_output",
        "call_id": "call_xxx",
        "output": "{\"temperature\":18,\"condition\":\"sunny\"}"
    }
}

conversation.item.retrieve

Description: Retrieves a conversation item stored on the server. Audio-type content in the response contains only the transcript (transcript), not the original audio data.

typestring(required)Event type. Fixed value: conversation.item.retrieve.item_idstring(required)ID of the conversation item to retrieve. The server returns the result in a conversation.item.retrieved event.

{
    "type": "conversation.item.retrieve",
    "item_id": "item_xxx"
}

conversation.item.delete

Description: Deletes a conversation item from the conversation context. The server confirms the deletion with a conversation.item.deleted event.

typestring(required)Event type. Fixed value: conversation.item.delete.item_idstring(required)ID of the conversation item to delete.

{
    "type": "conversation.item.delete",
    "item_id": "item_xxx"
}

response.create

Description: Triggers model inference. Behavior varies by mode:

Push-to-talk mode: Must be called manually. Commit the buffered audio with input_audio_buffer.commit first, or return a function_call_output result before triggering. Can't be called while a response is being generated.
server_vad mode: Typically triggered automatically by the server. Clients can also call it manually when no response is being generated. Can't be called while a response is being generated.
smart_turn mode: Can be called while waiting for the next user turn. Can't be called during an active turn (between input_audio_buffer.speech_started and response.done).

The optional response field overrides the session defaults for the current inference round. In Function Calling scenarios, after the client returns a function_call_output, this event triggers the second inference round.

In server_vad and smart_turn modes, manually triggered inferences can still be interrupted by new speech.

typestring(required)Event type. Fixed value: response.create.responseobject (optional)Overrides the session defaults for the current inference round. If not provided, the current session configuration is used.

Properties

modalitiesarray (optional)Overrides the output modalities for the current round. Valid values are the same as session.update modalities.voicestring (optional)Overrides the TTS voice for the current round.

{
    "type": "response.create",
    "response": {
        "modalities": ["audio", "text"]
    }
}

response.cancel

Description: Cancels the current inference. Any text generated so far is saved to the item list. The server then returns a response.done event with status=cancelled. An error is returned if no inference is in progress.

typestring(required)Event type. Fixed value: response.cancel.

{
    "type": "response.cancel"
}

​session.update

​input_audio_buffer.append

​input_audio_buffer.commit

​input_audio_buffer.clear

​conversation.item.create

​conversation.item.retrieve

​conversation.item.delete

​response.create

​response.cancel

session.update

input_audio_buffer.append

input_audio_buffer.commit

input_audio_buffer.clear

conversation.item.create

conversation.item.retrieve

conversation.item.delete

response.create

response.cancel