Client events for the Qwen-Omni-Realtime API.
session.update
After establishing a WebSocket connection, send this event first to update the default session configurations. The server validates the parameters upon receiving the session.update event. If the parameters are invalid, the server returns an error. If valid, the server applies the configuration and returns the updated configuration.
typestring(required)The event type. The value must be session.update. | |
sessionobject(optional)The session configuration.For Qwen3.8-Omni-Flash-Realtime, session is required when updating a session. The total input token limit is 196608, calculated using the same rules as Qwen3.5-Omni-Realtime.
Properties modalities array(optional)The model output modalities. Valid values:
string(optional)The voice for the model's audio output. For supported voices, see Voice list.Default voices:
object (optional)Input and output audio configuration. Omitted fields retain their existing defaults.
Properties input object (optional)User input audio configuration.
Properties format object (optional)Input format and sample rate. Configure these while the session is IDLE, before sending the first audio segment. They cannot change after audio input starts.
Properties type string (optional)For qwen3.5-omni-plus-realtime and qwen3.5-omni-flash-realtime: pcm (default, mono 16-bit raw PCM) or wav (mono 16-bit PCM in a WAV container). Multichannel input for Qwen3.8-Omni-Flash-Realtime requires pcm, which is the default.sample_rateinteger (optional)For qwen3.5-omni-plus-realtime and qwen3.5-omni-flash-realtime: 8000, 16000 (default), 24000, or 48000 Hz. Multichannel input for Qwen3.8-Omni-Flash-Realtime requires 16000 Hz, which is the default.The following four fields apply to Qwen3.8-Omni-Flash-Realtime multichannel input over WebSocket:sample_formatstring (optional)PCM sample format. Only s16le is supported; default: s16le.channelsinteger (optional)Input channel count: 1 (default), 2, or 4. Two-channel and four-channel spatial audio use twice the input tokens of ordinary audio. See Token calculation.packingstring (optional)Sample packing. Only interleaved is supported; default: interleaved.channel_layoutstring (optional)Default depends on channels: mono for one channel, raw_mic_array for two, and foa_ambix for four. An explicit value must match channels.Multichannel input requires PCM, 16000 Hz, s16le, and interleaved samples. Omitted fields use their defaults; supplied fields must have valid types and values. Do not send null instead of omitting a field.object (optional)Model output audio configuration.
Properties format object (optional)Output format and sample rate. Configure these at the start of the session, before audio interaction begins.
Properties type string (optional)For qwen3.5-omni-plus-realtime and qwen3.5-omni-flash-realtime: pcm (default, mono 16-bit raw PCM) or wav (mono 16-bit PCM in a WAV container).sample_rateinteger (optional)For qwen3.5-omni-plus-realtime and qwen3.5-omni-flash-realtime: 8000, 16000, 24000 (default), or 48000 Hz.string (optional)For Qwen3.8-Omni-Flash-Realtime, the output voice defaults to Tina. See the voice list for available voices and their voice values. session.voice remains available for existing integrations. For new integrations, use session.audio.output.voice. If both fields are set, the audio output uses the voice specified by session.audio.output.voice.object | null (optional)Input audio transcription configuration. Set to null to disable it.
Properties model string (required)The value is fixed to qwen3-asr-flash-realtime. Set {"model":"qwen3-asr-flash-realtime"} to enable transcription; the model value cannot be changed. The Python SDK's enable_input_audio_transcription is an update_session method argument, not a top-level session.update field.string(optional)The format of the user's input audio. Currently only supports pcm. The input audio must be a PCM audio stream at a 16 kHz sample rate.output_audio_formatstring(optional)The format of the model's output audio. Currently only supports pcm. The output audio is a PCM audio stream at a 24 kHz sample rate.smooth_outputboolean | null(optional)Applies only to qwen3-omni-flash-realtime-2025-09-15.Whether to enable a colloquial response style. Valid values:
string(optional)The system message that sets the model's role or objective.turn_detectionobject(optional)The Voice Activity Detection (VAD) configuration. Set to null to disable VAD and trigger model responses manually. If omitted, VAD is enabled with default parameters.
Properties type string(optional)The VAD type. Valid values:
float(optional)The VAD sensitivity. A lower value makes VAD more sensitive—more likely to detect faint sounds as speech, including background noise. A higher value requires clearer and louder speech to trigger.Valid range: [-1.0, 1.0]. Default: 0.5.silence_duration_msinteger(optional)The minimum silence duration after speech ends, in milliseconds. When exceeded, the model generates a response. A lower value speeds up responses but may cause false triggers during brief pauses.Default: 800. Valid range: [200, 6000].idle_timeout_msinteger(optional)Applies only to qwen3.5-omni-plus-realtime and qwen3.5-omni-flash-realtime models in server_vad mode.The idle timeout in milliseconds. After the server finishes audio playback and the user remains silent beyond this duration (no speech.started triggered), the model proactively generates a response to prompt the user to continue the conversation based on the current context. The timeout starts after the last model response audio finishes playing.Valid range: [5000, 30000].boolean(optional)Applies to Qwen3.8-Omni-Flash-Realtime and Qwen3.5-Omni-Realtime models.Whether to enable web search. Default: false. When enabled, the model decides autonomously whether to search to answer real-time questions.tools and enable_search are incompatible. Do not enable both at the same time.search_options object(optional)Web search configuration. Takes effect only when enable_search is enabled.
Properties enable_source boolean(optional)Whether to return the list of search result sources. Set to true to enable.object (optional)Video input configuration for Qwen3.8-Omni-Flash-Realtime.
Properties input object (optional)Video input parameters.
Properties representation_compact string (optional)Video representation aggregation. The initial session default is none, which preserves full fine-grained input. normal aggregates it to reduce computation when fine visual detail is unnecessary; for the same video input, it uses one quarter of the tokens used by none. See Token calculation. Set this before the first audio segment; it cannot change after audio input starts.array(optional)A list of tool definitions. The model decides whether to call a tool based on user input. For Qwen3.8-Omni-Flash-Realtime, Function Calling and MCP tools can be configured in the same session. Array elements are distinguished by type; tools and enable_search cannot both be enabled.
Function Calling (type=function) type string (required)Must be function.functionobject (required)The custom function definition.
Properties name string (required)The function name, matching the actual function name, such as get_current_weather or get_current_time.descriptionstring (optional)A description of what the function does. The model uses it to decide whether to call the tool.parametersobject (optional)The function's input parameters. The model uses this to extract parameter values. Omit this field if the function has no input.
Properties type string (required)Must be object.propertiesobject (optional)Defines each input parameter by name. In the example on the right, location is a custom parameter name; its type and description describe the data type and purpose.requiredarray (optional)Lists which parameters are required.
MCP (type=mcp) Required fields must be supplied when this MCP array element is present. Every addition or update requires both server_label and server_url; a label alone cannot reuse an existing configuration. Update MCP configuration only when there is no active Response. Connections, discovery, and calls are subject to quotas and timeouts. See MCP call limits.typestring (required)Must be mcp.server_labelstring (required)Unique MCP server label within the session. Length: 1–64 characters; letters, digits, underscores, and hyphens only.server_urlstring (required)MCP Streamable HTTP endpoint. Must be a public HTTPS address on port 443, up to 4096 characters, with no username, password, or fragment. Its domain must resolve to public addresses.authorizationstring (optional)Outbound Authorization value, up to 8192 printable ASCII characters.headersobject (optional)Additional outbound HTTP headers, up to 16 string-to-string pairs. Names must be 1–128 characters using standard HTTP header-name characters. Values must be printable ASCII and at most 8192 characters.Names cannot start with mcp-, proxy-, or x-forwarded-, or equal host, authorization, connection, content-length, transfer-encoding, accept, content-type, forwarded, cookie, origin, upgrade, te, or trailer.allowed_toolsarray of strings (optional)Filter applied after discovery. Omission allows all tools; an empty array exposes none. Each name is 1–64 characters, with letters, digits, underscores, periods, and hyphens only.require_approvalstring (optional)Approval policy: always (default) or never.server_url, authorization, and headers are used only for the server-side connection and are not echoed in session.updated. Do not reconstruct sensitive settings from that event. | |
temperaturefloat(optional)The sampling temperature, controlling output diversity. A higher value produces more varied content; a lower value produces more deterministic output.Valid range: [0, 2).Because temperature and top_p both control diversity, set only one.Default values:
| |
top_pfloat(optional)The nucleus sampling probability threshold, controlling output diversity. A higher value produces more varied content; a lower value produces more deterministic output.Valid range: (0, 1.0].Because temperature and top_p both control diversity, set only one.Default values:
| |
top_kinteger(optional)The candidate set size for sampling. For example, a value of 50 limits each generation step to the 50 highest-scoring tokens. A larger value increases randomness; a smaller value increases determinism. Set to null or a value greater than 100 to disable top_k, in which case only top_p applies.Minimum value: 0.Default values:
| |
max_tokensinteger(optional)The maximum number of tokens to return.TheFor qwen3.8-omni-flash-realtime, the valid range is [1, 65536]. For other models, the default and maximum values equal the model's maximum output length. See Model list.Use max_tokens to limit output length for tasks like summarization, keyword extraction, cost control, or latency reduction. | |
repetition_penaltyfloat(optional)Controls repetition in the model's output. A higher value reduces repetition; 1.0 means no penalty. qwen3.8-omni-flash-realtime also accepts 0. For other models, the value must be greater than 0.Default values:
| |
presence_penalty float(optional)Controls content repetition in the model's output.Valid range: [-2.0, 2.0]. A positive value reduces repetition; a negative value increases it.Default values:
| |
seedinteger(optional)Makes generation more deterministic. Pass the same seed value with unchanged parameters to get consistent results across runs.Valid range: 0 to 231−1. Default: -1. |
Tool configuration example
Replace the endpoint, credential, and tool name with your MCP server details.
Audio configuration examples
Two-channel configuration:
session.audio.output.voice example (uses the Long An Ling Xin voice, whose voice value is longanlingxin):
Video configuration example
Video aggregation example:
response.create
The response.create event instructs the server to generate a model response. In VAD mode, the server normally starts responses from speech activity; tool continuations follow the explicit flow below. For Function Calling, send this event after returning the tool result via conversation.item.create to continue generation. For Qwen3.8-Omni-Flash-Realtime MCP tools, the server executes the call; if a model continuation is needed, send one response.create after response.done for the Response containing this MCP call, without attaching MCP results.
The server replies with a response.created event, followed by one or more item and content events—such as conversation.item.created and response.content_part.added—and finally a response.done event.
typestring(required)The event type. The value must be response.create. |
response.cancel
Send this event to cancel an ongoing response. If no response is in progress, the server returns an error.
typestring(required)The event type. The value must be response.cancel. |
input_audio_buffer.append
Appends audio bytes to the input audio buffer.
typestring(required)The event type. The value must be input_audio_buffer.append. | |
audiostring(required)The Base64-encoded audio data. |
input_audio_buffer.commit
Commits the input audio buffer, creating a new user message item in the conversation. If the buffer is empty, the server returns an error.
- VAD mode: The client does not need to send this event. The server commits the audio buffer automatically.
- Manual mode: The client must commit the audio buffer to create a user message item.
input_audio_buffer.committed event.
If the client sent an input_image_buffer.append event, the input_audio_buffer.commit event also commits the image buffer.
typestring(required)The event type. The value must be input_audio_buffer.commit. |
input_audio_buffer.clear
Clears audio bytes from the buffer. The server replies with an input_audio_buffer.cleared event.
typestring(required)The event type. The value must be input_audio_buffer.clear. |
input_image_buffer.append
Adds image data to the image buffer. Images can come from local files or a live video stream.
The following limits apply to image inputs:
- Format: JPG or JPEG. Recommended resolution: 480p or 720p; maximum: 1080p.
- A single image after Base64 encoding must not exceed 256 KB. We recommend keeping the raw image size below 190 KB before encoding.
- Image data must be Base64-encoded.
- Recommended send frequency: 1 image per second.
- Send at least one
input_audio_buffer.appendevent before sending aninput_image_buffer.appendevent.
The image buffer is committed with the audio buffer using the input_audio_buffer.commit event.
typestring(required)The event type. The value must be input_image_buffer.append. | |
imagestring(required)The Base64-encoded image data. |
conversation.item.create
Send this event to create a conversation item, either a Function Calling result or, for Qwen3.8-Omni-Flash-Realtime, an MCP approval response. For Function Calling, run the tool locally, return its result as function_call_output, then send response.create. The server executes MCP tools; use mcp_approval_response for approval and follow the approval response continuation sequence.
event_idstring (optional)A client-generated event ID for tracing.typestring(required)The event type. The value must be conversation.item.create. | |
itemobject(required)The conversation item to create. Cannot be empty.
Properties id string (optional)The conversation item ID. Specify one to align with local state; otherwise the server generates one.typestring (required)Use function_call_output for a Function Calling result or mcp_approval_response for a Qwen3.8-Omni-Flash-Realtime MCP approval reply.call_idstring (conditionally required)Required only for function_call_output. Use the call_id returned in response.function_call_arguments.done.outputstring (conditionally required)Required only for function_call_output. The tool function execution result.approval_request_idstring (conditionally required)Required only for mcp_approval_response. Must exactly match the item.id of an unhandled mcp_approval_request.approveboolean (conditionally required)Required only for mcp_approval_response. true allows execution; false rejects the call, which enters failed. |
MCP approval response (Qwen3.8-Omni-Flash-Realtime)
When require_approval is always or omitted, the server may send mcp_approval_request. Reply using the conversation.item.create fields above with item.type="mcp_approval_response". The server executes the MCP tool. If a model continuation is needed, wait for response.done for the Response containing this MCP call, then send one response.create without MCP results.
Approval request IDs, item IDs, and event IDs returned by the server are opaque strings. Do not rely on their lengths, prefixes, or generation rules.
See Qwen-Omni-Realtime .