Skip to main content
Alibaba Cloud Model Studio home pageAlibaba Cloud Model Studio
Real-time audio and video translation (Qwen-Livetranslate-Realtime)

Server events

Server-side events for the qwen3.5-livetranslate-flash-realtime API.

Reference: Real-time audio and video translation - Qwen

error

Error message returned by the server.
event_idstringThe unique identifier for this event.typestringThe event type. The value is always error.errorobjectDetailed information about the error.

Properties

typestringThe error type.codestringThe error code.messagestringThe error message.paramstringThe parameter that is related to the error, such as session.modalities.
{
  "event_id": "event_RoUu4T8yExPMI37GKwaOC",
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "code": "invalid_value",
    "message": "Invalid modalities: ['audio']. Supported combinations are: ['text'] and ['audio', 'text'].",
    "param": "session.modalities"
  }
}

session.created

When a client connects, the server returns this event first with the default session configurations.
event_idstringThe unique identifier for this event.typestringThe event type. The value is always session.created.sessionobjectThe session configuration.

Properties

idstringThe unique identifier for the session.objectstringThe value is always realtime.session.modelstringThe model in use.modalitiesarrayThe output modality settings for the model.voicestringThe voice for the audio generated by the model.input_audio_formatstringThe format of the input audio. Defaults to pcm.sample_rateintegerThe sample rate of the input audio, in Hz.output_audio_formatstringThe format of the output audio. Defaults to pcm.turn_detectionobjectThe VAD (Voice Activity Detection) configuration.translationobject(Optional)The translation configuration.

Properties

language string (Optional)The target language for translation.corpusobject(Optional)Custom terminology configuration for improving translation accuracy of domain-specific terms.

Properties

corpus.phrasesobject(Optional)A mapping of source-language terms to target-language translations. For details, see Supported languages.
{
    "event_id": "event_QxBGpjBDmDDQQWDtrqBKB",
    "type": "session.created",
    "session": {
        "id": "sess_OozZ1vtbPt2muDflHODIH",
        "object": "realtime.session",
        "model": "qwen3.5-livetranslate-flash-realtime",
        "modalities": [
            "text",
            "audio"
        ],
        "voice": "Cherry",
        "input_audio_format": "pcm",
        "sample_rate": 16000,
        "output_audio_format": "pcm",
        "turn_detection": {
            "type": "server_vad",
            "threshold": 0.2,
            "silence_duration_ms": 1000
        },
        "translation": {
           "language": "en",
           "corpus": {
               "phrases": {
                   "人工智能": "Artificial Intelligence",
                   "机器学习": "Machine Learning"
               }
           }
        }
    }
}

session.updated

Returned after a successful session.update request. If the request fails, an error event is returned instead.
event_idstringThe unique identifier for this event.typestringThe event type. The value is always session.updated.sessionobjectThe session configuration.

Properties

idstringThe unique identifier for the session.objectstringThe value is always realtime.session.modelstringThe model in use.modalitiesarrayThe output modality settings for the model.voicestringThe voice for the audio generated by the model.sample_rateinteger(Optional)The sample rate of the input audio, in Hz.input_audio_formatstringThe format of the input audio. The value is always pcm.output_audio_formatstringThe format of the output audio. The value is always pcm.input_audio_transcriptionobjectThe input audio transcription configuration. Returned only when the input_audio_transcription.model parameter is configured in the session.

Properties

modelstringThe speech recognition model.languagestringThe configured speech recognition language.
turn_detectionobjectThe VAD (Voice Activity Detection) configuration. This field is not returned in Manual mode (when the client sets this parameter to null in session.update).

Properties

typestringThe VAD type. The value is always server_vad.thresholdfloatThe VAD detection sensitivity.prefix_padding_msintegerThe duration of audio (in milliseconds) retained before speech starts, to avoid losing the beginning of speech.silence_duration_msintegerThe minimum duration of silence (in milliseconds) required after speech ends to determine that speech has concluded.create_responsebooleanWhether to automatically trigger a translation response after VAD detects that speech has ended.interrupt_responsebooleanWhether to interrupt the current translation response being generated when VAD detects a new speech turn.
translationobject(Optional)The translation configuration.

Properties

language string (Optional)The target language for translation.corpusobject(Optional)Custom terminology configuration for improving translation accuracy of domain-specific terms.

Properties

corpus.phrasesobject(Optional)A mapping of source-language terms to target-language translations.
enable_voice_clonebooleanWhether to enable voice cloning.voice_clone_optionsobjectVoice cloning control parameters. Returned only when enable_voice_clone is true.

Properties

frequencystringThe voice cloning frequency.
{
    "event_id": "event_QxBGpjBDmDDQQWDtrqBKB",
    "type": "session.updated",
    "session": {
        "id": "sess_OozZ1vtbPt2muDflHODIH",
        "object": "realtime.session",
        "model": "qwen3.5-livetranslate-flash-realtime",
        "modalities": [
            "text",
            "audio"
        ],
        "voice": "Ethan",
        "sample_rate": 16000,
        "input_audio_format": "pcm",
        "output_audio_format": "pcm",
        "input_audio_transcription": {
            "model": "qwen3-asr-flash-realtime",
            "language": "zh"
        },
        "turn_detection": {
            "type": "server_vad",
            "threshold": 0.2,
            "prefix_padding_ms": 300,
            "silence_duration_ms": 1000,
            "create_response": true,
            "interrupt_response": true
        },
        "translation": {
           "language": "en",
           "corpus": {
               "phrases": {
                   "人工智能": "Artificial Intelligence",
                   "机器学习": "Machine Learning"
               }
           }
        },
        "enable_voice_clone": true,
        "voice_clone_options": {
            "frequency": "once"
        }
    }
}

session.finished

Session is finished and all audio translations are complete. Sent only after the client sends a session.finish request. The client can then disconnect.
event_idstringThe unique identifier for this event.typestringThe event type. The value is always session.finished.
{
    "event_id": "event_xxx",
    "type": "session.finished"
}

response.created

Returned when the server generates a new model response.
event_idstringThe unique identifier for this event.typestringThe event type. The value is always response.created.responseobjectThe response object.

Properties

id stringThe unique identifier for the response.conversation_id stringThe unique identifier for the current session.object stringThe object type. For this event, the value is always realtime.response.status stringThe response status. Valid values:
  • completed
  • failed
  • in_progress
  • incomplete
modalities arrayResponse modality.voice stringThe voice of the generated audio.output_audio_format stringThe format of the output audio.output arrayThis event is currently empty.
{
    "event_id": "event_L8hHVI5jYis6BzAjnPWJh",
    "type": "response.created",
    "response": {
        "id": "resp_P79OOMs8LnrXVpiIHUCKR",
        "object": "realtime.response",
        "conversation_id": "conv_UFClXtYkRkFXrs48y8pmK",
        "status": "in_progress",
        "modalities": [
            "text",
            "audio"
        ],
        "voice": "Cherry",
        "output_audio_format": "pcm16",
        "output": []
    }
}

response.done

Returned after response generation is complete. Contains all output items except raw audio data.
event_id stringThe unique identifier for this event.type stringThe event type. The value is always response.done.response objectThe response object.

Properties

id stringThe unique identifier for the response.conversation_id stringThe unique identifier for the current session.object stringThe object type. For this event, the value is always realtime.response.status stringThe status of the response.modalities arrayThe modality of the response.voice stringThe voice used for the audio generated by the model.output_audio_format stringThe format of the output audio.output objectThe output of the response.

Properties

id stringThe unique identifier for the response output.type stringThe type of the output item. The value is currently always message.object stringThe object type of the output item. The value is currently always realtime.item.status stringThe status of the output item.role stringThe role of the output item.content arrayThe content of the output item.

Properties

type stringThe type of the output content. The value is text for plain text output and audio when the output includes audio.text stringThe text content of the output.transcript stringThe text transcription of the audio content.
usage objectThe token consumption information for this response.
{
  "event_id": "event_CNea8oXNipVanSg2VIzkO",
  "type": "response.done",
  "response": {
    "id": "resp_TfhYTqej692vsGA2jNEtH",
    "object": "realtime.response",
    "conversation_id": "conv_ZtyLfKVm8XqLwYRlsuDih",
    "status": "completed",
    "modalities": [
      "text",
      "audio"
    ],
    "voice": "Cherry",
    "output_audio_format": "pcm16",
    "output": [
      {
        "id": "item_MKtkMwN9RtcyE9eJShyWy",
        "object": "realtime.item",
        "type": "message",
        "status": "completed",
        "role": "assistant",
        "content": [
          {
            "type": "audio",
            "transcript": "Hello? "
          }
        ]
      }
    ],
    "usage": {
      "total_tokens": 56,
      "input_tokens": 47,
      "output_tokens": 9,
      "input_tokens_details": {
        "text_tokens": 20,
        "audio_tokens": 27
      },
      "output_tokens_details": {
        "text_tokens": 2,
        "audio_tokens": 7
      }
    }
  }
}

response.text.text

Returned for text-only output when the model generates text incrementally.
event_idstringA unique identifier for the event.typestringThe type of the event. The value is always response.text.text.textstringThe incremental text that is returned.response_idstringThe response ID.item_idstringA unique identifier for the message item.output_indexintegerCurrently, the value is always 0.content_indexintegerCurrently, the value is always 0.stashstringTemporary text generated by the model. Concatenate stash with the current text to form an interim result. The system updates text and stash continuously using response.text.text events until it receives a response.text.done event. The final complete text is then available in the text field.
{
    "event_id": "event_B1lIeyOXR7qJMEExbqtTG",
    "type": "response.text.text",
    "response_id": "resp_B1lIdtjF4Noqpn5NOjznj",
    "item_id": "item_B1lIdJsAJlJiFs8ztWpJt",
    "output_index": 0,
    "content_index": 0,
    "text": "How are",
    "stash": " you today?"
}

response.text.done

Returned when text generation finishes for text-only output, or if the response is interrupted, incomplete, or canceled.
event_idstringThe unique identifier for this event.typestringThe event type. The value is always response.text.done.response_idstringThe unique identifier for the response.item_idstringThe unique identifier for the message item.output_indexintegerThe value is currently always 0.content_indexintegerThe value is currently always 0.text stringThe complete text output from the model.
{
    "event_id": "event_B1lIeE2Nac33zn5V7h2mm",
    "type": "response.text.done",
    "response_id": "resp_B1lIdtjF4Noqpn5NOjznj",
    "item_id": "item_B1lIdJsAJlJiFs8ztWpJt",
    "output_index": 0,
    "content_index": 0,
    "text": "How can I assist you today?"
}

response.audio.delta

Returned when audio output is enabled and the model generates audio incrementally.
event_idstringA unique identifier for the event.typestringThe event type. The value is always response.audio.delta.response_idstringA unique identifier for the response.item_idstringA unique identifier for the message item.output_index integerThe value is always 0.content_index integerThe value is always 0.delta stringThe incremental audio data that is output by the model. The data is Base64-encoded.
{
    "event_id": "event_B1osWMZBtrEQbiIwW0qHQ",
    "type": "response.audio.delta",
    "response_id": "resp_P79OOMs8LnrXVpiIHUCKR",
    "item_id": "item_OFaPGtzfWCPyGzxnuEX9i",
    "output_index": 0,
    "content_index": 0,
    "delta": "UklGRnoGAABXQVZFZm10IBAAAAAB..."
}

response.audio.done

Returned when audio generation is complete. Also returned if the response is interrupted, incomplete, or canceled. Does not contain complete audio data.
event_idstringThe unique identifier for this event.typestringThe event type. This is always response.audio.done.response_idstringThe unique identifier for the response.item_idstringThe unique identifier for the message item.output_indexintegerThe value is always 0.content_indexintegerThe value is always 0.
{
    "event_id": "event_B1osWMWoDRYyITDyNYcBu",
    "type": "response.audio.done",
    "response_id": "resp_P79OOMs8LnrXVpiIHUCKR",
    "item_id": "item_OFaPGtzfWCPyGzxnuEX9i",
    "output_index": 0,
    "content_index": 0
}

input_audio_buffer.speech_started

Returned when the server-side VAD (Voice Activity Detection) detects that the user has started speaking.
event_idstringThe unique identifier for this event.typestringThe event type. The value is always input_audio_buffer.speech_started.audio_start_msintegerThe time point (in milliseconds) when speech was detected to start, as an offset from the beginning of the audio stream.item_idstringThe unique identifier for the associated message item.
{
    "event_id": "event_xxx",
    "type": "input_audio_buffer.speech_started",
    "audio_start_ms": 568,
    "item_id": "item_xxx"
}

input_audio_buffer.speech_stopped

Returned when the server-side VAD detects that the user has stopped speaking, indicating the end of the current speech input turn. Because the translation response is generated synchronously with the streaming speech input, the actual translation response may have already started during the speech input process, without needing to wait for this event.
event_idstringThe unique identifier for this event.typestringThe event type. The value is always input_audio_buffer.speech_stopped.audio_end_msintegerThe time point (in milliseconds) when speech was detected to end, as an offset from the beginning of the audio stream.item_idstringThe unique identifier for the associated message item.
{
    "event_id": "event_xxx",
    "type": "input_audio_buffer.speech_stopped",
    "audio_end_ms": 3900,
    "item_id": "item_xxx"
}

input_audio_buffer.committed

In Manual mode (turn_detection is null), after the client sends an input_audio_buffer.commit event, the server returns this event as confirmation and automatically starts generating the translation response.
event_idstringThe unique identifier for this event.typestringThe event type. The value is always input_audio_buffer.committed.
{
    "event_id": "event_xxx",
    "type": "input_audio_buffer.committed"
}

input_audio_buffer.cleared

After the client sends an input_audio_buffer.clear event, the server returns this event as confirmation that the uncommitted audio data in the buffer has been cleared.
event_idstringThe unique identifier for this event.typestringThe event type. The value is always input_audio_buffer.cleared.
{
    "event_id": "event_xxx",
    "type": "input_audio_buffer.cleared"
}

conversation.item.created

Returned when a new message item is created in the conversation. This event is triggered in the following scenarios:
  • When the server starts generating a translation response, it creates a corresponding assistant message item (at this point, content is an empty array, and the content is progressively populated with the streaming response).
  • In Manual mode, after the client sends an input_audio_buffer.commit event, the server additionally creates a message item corresponding to the user's input audio (content contains {"type": "input_audio"}).
For the same VAD segment, the server creates separate message items for the speech recognition result and the translation result. The item.id of the speech recognition result matches the previous_item_id in the translation result event. Use these values to associate and display the source text with its translation.
event_idstringThe unique identifier for this event.typestringThe event type. The value is always conversation.item.created.previous_item_idstringThe unique identifier of the previous message item. For a translation result event, this value matches the item.id of the speech recognition result for the same VAD segment.itemobjectThe message item information.

Properties

idstringThe unique identifier for the message item.typestringThe value is always message.objectstringThe value is always realtime.item.statusstringThe status of the message item.rolestringThe role of the message. Valid values: assistant or user.contentarrayThe content of the message. When a response is first created, this is an empty array that is progressively populated with the streaming response. For user message items created by commit in Manual mode, it contains {"type": "input_audio"}.
{
    "event_id": "event_xxx",
    "type": "conversation.item.created",
    "previous_item_id": "item_asr_xxx",
    "item": {
        "id": "item_translation_xxx",
        "object": "realtime.item",
        "type": "message",
        "status": "in_progress",
        "role": "assistant",
        "content": []
    }
}

conversation.item.input_audio_transcription.text

When input_audio_transcription.model is configured, the server streams speech recognition results in the original source language.
event_idstringThe unique identifier for this event.typestringThe event type. The value is always conversation.item.input_audio_transcription.text.item_idstringThe unique identifier for the message item.content_indexintegerThe value is currently always 0.textstringThe confirmed recognition text.stashstringThe recognition text that is pending confirmation. This text may be corrected by subsequent events.languagestringThe detected source language.emotionstringThe detected emotion of the recognized audio. Supported emotions are as follows:
  • surprised: Surprised
  • neutral: Neutral
  • happy: Happy
  • sad: Sad
  • disgusted: Disgusted
  • angry: Angry
  • fearful: Fearful
{
    "event_id": "event_xxx",
    "type": "conversation.item.input_audio_transcription.text",
    "item_id": "item_xxx",
    "content_index": 0,
    "text": "",
    "stash": "The weather is really nice today",
    "language": "zh",
    "emotion": "neutral"
}

conversation.item.input_audio_transcription.completed

When input_audio_transcription.model is configured, returns the final recognition result after speech recognition completes.
event_idstringThe unique identifier for this event.typestringThe event type. This is always conversation.item.input_audio_transcription.completed.item_idstringThe unique identifier for the message item.content_indexintegerThis is currently always 0.transcriptstringThe complete speech recognition result in the original source language.languagestringThe detected source language.emotionstringThe detected emotion of the recognized audio. Supported emotions:
  • surprised: surprised
  • neutral: calm/neutral
  • happy: happy
  • sad: sad
  • disgusted: disgusted
  • angry: angry
  • fearful: fearful
{
    "event_id": "event_xxx",
    "type": "conversation.item.input_audio_transcription.completed",
    "item_id": "item_xxx",
    "content_index": 0,
    "transcript": "What a beautiful day! Let's go for a walk in the park together.",
    "language": "en",
    "emotion": ""
}

conversation.item.input_audio_transcription.failed

Returned when audio input was received but speech recognition failed. This event is handled separately from other error events, making it easier for the client to identify the specific associated item.
typestringThe event type. The value is always conversation.item.input_audio_transcription.failed.item_idstringThe ID of the associated conversation item.content_indexintegerThe index of the content part that contains the audio.error.codestringThe error code.error.messagestringThe error message.
{
    "event_id": "event_xxx",
    "type": "conversation.item.input_audio_transcription.failed",
    "item_id": "item_xxx",
    "content_index": 0,
    "error": {
        "code": "xxx",
        "message": "xxx",
        "param": "xxx"
    }
}

response.audio_transcript.text

Returned for audio output to display real-time translation.
event_idstringThe unique identifier for this event.typestringThe type of the event. The value is always response.audio_transcript.text.response_idstringThe unique identifier for the response.item_idstringThe unique identifier for the message item.output_index integerCurrently, the value is always 0.content_index integerCurrently, the value is always 0.textstringThe confirmed translation text segment.stashstringTemporary translation text, concatenated with text to form interim results. The system updates text and stash continuously via response.audio_transcript.text events until a response.audio_transcript.done event is received. Then retrieve the final translation from transcript.
{
  "event_id": "event_xxx",
  "type": "response.audio_transcript.text",
  "response_id": "resp_xxx",
  "item_id": "item_xxx",
  "output_index": 0,
  "content_index": 0,
  "text": "Hello,",
  "stash": " who are you?"
}

response.audio_transcript.done

Returned when audio output is enabled and text generation finishes.
event_idstringThe unique identifier for this event.typestringThe event type. This is always response.audio_transcript.done.response_idstringThe unique identifier for the response.item_idstringThe unique identifier for the message item.output_index integerThis is currently always 0.content_index integerThis is currently always 0.transcriptstringThe complete text.
{
    "event_id": "event_VN4Q4GJugLcc1S23viW8E",
    "type": "response.audio_transcript.done",
    "response_id": "resp_P79OOMs8LnrXVpiIHUCKR",
    "item_id": "item_JvJauNH2CTXb1D9WV6pD4",
    "output_index": 0,
    "content_index": 0,
    "transcript": "How can I assist you today?"
}

response.output_item.added

Returned when a new output item is added during response generation.
event_idstringThe unique identifier for this event.typestringThe event type. The value is always response.output_item.added.response_idstringThe unique identifier for the response.output_index integerThe value is currently always 0.item objectInformation about the output item.

Properties

idstringThe unique identifier for the output item.typestringThe value is always message.objectstringThe value is always realtime.item.statusstringThe status of the output item.rolestringThe role of the message.contentstringThe content of the message.
{
    "event_id": "event_B4O5yPt3Gjnjy5eYH3plG",
    "type": "response.output_item.added",
    "response_id": "resp_P79OOMs8LnrXVpiIHUCKR",
    "output_index": 0,
    "item": {
        "id": "item_OFaPGtzfWCPyGzxnuEX9i",
        "object": "realtime.item",
        "type": "message",
        "status": "in_progress",
        "role": "assistant",
        "content": []
    }
}

response.output_item.done

Returned when an output item is completed.
event_idstringThe unique identifier for this event.typestringThe event type. The value is always response.output_item.done.response_idstringThe unique identifier for the response.output_indexintegerThe value is currently always 0.itemobjectInformation about the output item.

Properties

idstringThe unique identifier for the output item.objectstringThe value is always realtime.item.typestringThe value is always message.statusstringThe status of the output item.rolestringThe role of the message sender.contentstringThe content of the message.
{
    "event_id": "event_XkiwbYTBC9Wcdwy6uYJ2G",
    "type": "response.output_item.done",
    "response_id": "resp_P79OOMs8LnrXVpiIHUCKR",
    "output_index": 0,
    "item": {
        "id": "item_JvJauNH2CTXb1D9WV6pD4",
        "object": "realtime.item",
        "type": "message",
        "status": "completed",
        "role": "assistant",
        "content": [
            {
                "type": "audio",
                "text": "Hello, I am a large language model developed by Alibaba Cloud. My name is Qwen. How can I help you?"
            }
        ]
    }
}

response.content_part.added

Returned when a new content part is added.
event_id stringThe unique ID of the event.type stringThe type of the event. The value is always response.content_part.added.response_id stringThe unique ID of the response.item_id stringThe unique ID of the message item.output_index integerThe value is always 0.content_index integerThe value is always 0.part objectOutputs item information.

Properties

type stringThe type of the content part.text stringThe text of the content part.
{
    "event_id": "event_J2UixwYKZsXg7c9YXZetL",
    "type": "response.content_part.added",
    "response_id": "resp_P79OOMs8LnrXVpiIHUCKR",
    "item_id": "item_OFaPGtzfWCPyGzxnuEX9i",
    "output_index": 0,
    "content_index": 0,
    "part": {
        "type": "audio",
        "text": ""
    }
}

response.content_part.done

Returned when a content part is completed.
event_idstringThe unique identifier for this event.typestringThe event type. This is always response.content_part.done.response_idstringThe unique identifier for the response.item_idstringThe unique identifier for the message item.output_index integerThe value is always 0.content_index integerThe value is always 0.part objectInformation about the content part.

Properties

typestringThe type of the content part.textstringThe text of the content part.
{
    "event_id": "event_VN4Q4GJugLcc1S23viW8E",
    "type": "response.content_part.done",
    "response_id": "resp_P79OOMs8LnrXVpiIHUCKR",
    "item_id": "item_JvJauNH2CTXb1D9WV6pD4",
    "output_index": 0,
    "content_index": 0,
    "part": {
        "type": "audio",
        "text": "Hello, I am a large language model developed by Alibaba Cloud. My name is Qwen. How can I help you?"
    }
}
Alibaba Cloud Model Studio home pageAlibaba Cloud Model Studio
API Usage
Text Generation
Image Generation
  • FAQ
Video Generation
Audio
Real-time Multimodal
Realtime API
Text Embedding
More Models
Toolkit/Framework
Model Production
File Management
More