Skip to main content
Alibaba Cloud Model Studio home pageAlibaba Cloud Model Studio
Text generation

Streaming output

In real-time chat or long-text generation applications, long wait times degrade user experience and may trigger server-side timeouts, causing tasks to fail. Streaming output addresses these issues by continuously returning fragments of text as the model generates them.

How it works

Streaming output uses the Server-Sent Events (SSE) protocol. After a streaming request starts, the server establishes an HTTP persistent connection with the client. Each time the model generates a text block (called a chunk), it immediately pushes it through this connection. Once all content is generated, the server sends an end signal. The client listens to the event stream and receives and processes text chunks in real time—for example, rendering characters one by one on the interface. This contrasts with non-streaming calls, which return all content at once.
The components above are for reference only and do not send actual requests.

Billing

Streaming output uses the same billing rule as non-streaming calls, charging based on the number of input tokens and output tokens in the request. If a request is interrupted, output tokens are counted only for the portion generated before the server received the termination request.

How to use

Qwen3 open-source edition, QwQ commercial and open-source editions, QVQ, and Qwen-Omni support only streaming output.

Step 1: Configure your API key and select a region

You must have obtained an API key and configured it as an environment variable.
Configuring your API key as an environment variable (DASHSCOPE_API_KEY) is more secure than hard coding it in your code.

Step 2: Make a streaming request

  • OpenAI compatible
  • DashScope
  • How to enable Set stream to true.
  • View token usage The OpenAI protocol does not return token usage by default. Set stream_options={"include_usage": true} so the last returned data chunk includes token usage information.
  • Python
  • Node.js
  • curl
import os
from openai import OpenAI

# 1. Prepare: Initialize the client
client = OpenAI(
    # Configure the API key using an environment variable to avoid hard coding.
    api_key=os.environ["DASHSCOPE_API_KEY"],
    # The API key is tightly bound to a region. Ensure base_url matches the region of your API key.
    # Singapore region URL. Replace {WorkspaceId} with your actual workspace ID. URLs vary by region.
    base_url="https://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/compatible-mode/v1",
)

# 2. Make a streaming request
completion = client.chat.completions.create(
    model="qwen-plus",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Please introduce yourself"}
    ],
    stream=True,
    stream_options={"include_usage": True}
)

# 3. Handle the streaming response
# Store response fragments in a list. Joining them at the end is more efficient than repeated string concatenation.
content_parts = []
print("AI: ", end="", flush=True)

for chunk in completion:
    if chunk.choices:
        content = chunk.choices[0].delta.content or ""
        print(content, end="", flush=True)
        content_parts.append(content)
    elif chunk.usage:
        print("\n--- Request usage ---")
        print(f"Input Tokens: {chunk.usage.prompt_tokens}")
        print(f"Output Tokens: {chunk.usage.completion_tokens}")
        print(f"Total Tokens: {chunk.usage.total_tokens}")

full_response = "".join(content_parts)
# print(f"\n--- Full response ---\n{full_response}")

Response 

AI: Hello! I am Qwen, a large-scale language model independently developed by Tongyi Lab under Alibaba Group. I can answer questions, create content such as stories, official documents, emails, scripts, perform logical reasoning, programming, express opinions, play games, and more. I support multiple languages, including but not limited to Chinese, English, German, French, and Spanish. If you have any questions or need help, feel free to ask me anytime!
--- Request usage ---
Input Tokens: 26
Output Tokens: 87
Total Tokens: 113

Streaming output for multimodal models

Multimodal models support adding images, audio, and other content to conversations. Their streaming output implementation differs from text-only models in the following ways:
  • User message construction: Multimodal model inputs include not only text but also images, audio, and other multimodal information.
  • DashScope SDK interface: Use the MultiModalConversation interface in the DashScope Python SDK. Use the MultiModalConversation class in the DashScope Java SDK.
For multimodal models, see Image and video understanding, Text extraction, Audio understanding—Qwen3-Omni-Captioner, Kimi, etc. The Qwen-Omni model supports only streaming output because its output can include text or audio and other multimodal content. Its result parsing differs from other models. For details, see Omni-modal.
  • OpenAI compatible
  • DashScope
Python
from openai import OpenAI
import os

client = OpenAI(
    # API keys differ by region. Get your API key: https://www.alibabacloud.com/help/zh/model-studio/get-api-key
    # If you haven't configured an environment variable, replace the next line with your Model Studio API key: api_key="sk-xxx"
    api_key=os.getenv("DASHSCOPE_API_KEY"),
    # Singapore region URL. Replace {WorkspaceId} with your actual workspace ID. URLs vary by region.
    base_url="https://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/compatible-mode/v1",
)

completion = client.chat.completions.create(
    model="qwen3-vl-plus",  # Replace with other multimodal models as needed and adjust messages accordingly
    messages=[
        {"role": "user",
        "content": [{"type": "image_url",
                    "image_url": {"url": "https://dashscope.oss-cn-beijing.aliyuncs.com/images/dog_and_girl.jpeg"},},
                    {"type": "text", "text": "What scene is depicted in the image?"}]}],
    stream=True,
  # stream_options={"include_usage": True}
)
full_content = ""
print("Streaming output content:")
for chunk in completion:
    # If stream_options.include_usage is True, the last chunk's choices field is an empty list and should be skipped (token usage can be obtained via chunk.usage)
    if chunk.choices and chunk.choices[0].delta.content != "":
        full_content += chunk.choices[0].delta.content
        print(chunk.choices[0].delta.content)
print(f"Full content: {full_content}")

Streaming output for thinking models

Thinking models first return reasoning_content (the thought process), then return content (the response). Determine whether the current stage is thinking or responding based on the data packet status.
For details on thinking models, see Deep thinking, Image and video understanding, Visual reasoning.
For streaming output implementation of Qwen3-Omni-Flash (thinking mode), see Omni-modal.
  • OpenAI compatible
  • DashScope
Below is the response format when calling the thinking mode of the qwen-plus model using the OpenAI Python SDK in streaming mode:
# Thinking stage
...
ChoiceDelta(content=None, function_call=None, refusal=None, role=None, tool_calls=None, reasoning_content='Cover all key points while')
ChoiceDelta(content=None, function_call=None, refusal=None, role=None, tool_calls=None, reasoning_content='remaining natural and fluent.')
# Response stage
ChoiceDelta(content='Hello! I am **Qwen', function_call=None, refusal=None, role=None, tool_calls=None, reasoning_content=None)
ChoiceDelta(content='** (', function_call=None, refusal=None, role=None, tool_calls=None, reasoning_content=None)
...
  • If reasoning_content is not None and content is None, the current stage is thinking.
  • If reasoning_content is None and content is not None, the current stage is responding.
  • If both are None, the stage remains the same as the previous packet.
  • Python
  • Node.js
  • HTTP

Example code 

from openai import OpenAI
import os

# Initialize the OpenAI client
client = OpenAI(
    # If you haven't configured an environment variable, replace with your Alibaba Cloud Model Studio API key: api_key="sk-xxx"
    api_key=os.getenv("DASHSCOPE_API_KEY"),
    base_url="https://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/compatible-mode/v1",
)

messages = [{"role": "user", "content": "Who are you"}]

completion = client.chat.completions.create(
    model="qwen-plus",  # Replace with other deep-thinking models as needed
    messages=messages,
    # The enable_thinking parameter enables the thinking process. This parameter has no effect on models qwen3-30b-a3b-thinking-2507, qwen3-235b-a22b-thinking-2507, and QwQ.
    extra_body={"enable_thinking": True},
    stream=True,
    # stream_options={
    #     "include_usage": True
    # },
)

reasoning_content = ""  # Full thought process
answer_content = ""  # Full response
is_answering = False  # Whether in the response stage
print("\n" + "=" * 20 + "Thought process" + "=" * 20 + "\n")

for chunk in completion:
    if not chunk.choices:
        print("\nUsage:")
        print(chunk.usage)
        continue

    delta = chunk.choices[0].delta

    # Collect only thinking content
    if hasattr(delta, "reasoning_content") and delta.reasoning_content is not None:
        if not is_answering:
            print(delta.reasoning_content, end="", flush=True)
        reasoning_content += delta.reasoning_content

    # Received content, start responding
    if hasattr(delta, "content") and delta.content:
        if not is_answering:
            print("\n" + "=" * 20 + "Full response" + "=" * 20 + "\n")
            is_answering = True
        print(delta.content, end="", flush=True)
        answer_content += delta.content

Response 

====================Thought process====================

Okay, the user asked "Who are you," so I need to give an accurate and friendly answer. First, I should confirm my identity as Qwen, developed by Tongyi Lab under Alibaba Group. Next, explain my main functions, like answering questions, creating text, logical reasoning, etc. Keep the tone approachable and avoid overly technical terms so the user feels comfortable. Also, avoid complex jargon and ensure the answer is concise. Additionally, include some interactive elements to encourage further questions. Finally, check for any missing key information, such as my Chinese name "Tongyi Qianwen" and English name "Qwen," along with my company and lab. Make sure the response is comprehensive and meets user expectations.
====================Full response====================

Hello! I am Qwen, a large-scale language model independently developed by Tongyi Lab under Alibaba Group. I can answer questions, create text, perform logical reasoning, programming, and more, aiming to provide high-quality information and services. You can call me Qwen or simply Tongyi Qianwen. How can I help you?

Going live

  • Performance and resource management: In backend services, maintaining an HTTP persistent connection for each streaming request consumes resources. Configure your service with appropriate connection pool size and timeout values. In high concurrency scenarios, monitor file descriptor usage to prevent exhaustion.
  • Client-side rendering: On web frontends, use the ReadableStream and TextDecoderStream APIs to smoothly handle and render SSE event streams, delivering the best user experience.
  • Model monitoring:
    • Key metrics: Monitor Time to First Token (TTFT), the core metric for streaming experience. Also monitor API error rate and average response time.
    • Alerting: Set alerts for abnormal API error rates, especially 4xx and 5xx errors.
  • Nginx proxy configuration: If using Nginx as a reverse proxy, its default output buffering (proxy_buffering) breaks the real-time nature of streaming responses. To ensure data is pushed to clients immediately, disable this feature by setting proxy_buffering off in your Nginx configuration file.

Error codes

If the model call fails and returns an error message, see Error codes for resolution.

FAQ

Q: Why is there no usage information in the response?

A: The OpenAI protocol does not return usage information by default. Set the stream_options parameter to include usage information in the final returned packet.

Q: Does enabling streaming output affect the model's response quality?

A: No. However, some models support only streaming output, and non-streaming calls might cause timeout errors. We recommend using streaming output.

Q: What is the difference between non-streaming and streaming calls?

A: Key differences:
  • Timeout limit: For non-streaming calls, the maximum timeout is at least 300 seconds and varies by region and model. If not completed in time, the request is terminated.
  • Output structure: Non-streaming calls return the complete response (a single JSON object) at once. Streaming calls return data chunks progressively via the SSE protocol, with each chunk containing part of the generated content. The client must assemble these chunks.
  • Feature compatibility: Both support features like JSON Mode and Function Call with no functional differences.
We recommend using streaming output to avoid timeouts and improve user experience.

Q: Does streaming output support JSON Mode (structured output)?

A: Yes. Set stream to true and response_format to {"type": "json_object"} in the request. The model will return JSON-formatted content fragments progressively. The final assembled output will be valid JSON.
Alibaba Cloud Model Studio home pageAlibaba Cloud Model Studio
Get Started
Billing
Token Plan
Model Playground
Clients and Developer Tools
Model Inference
Fine-tuning
Deployment
Evaluation
Statistics and Monitoring
Security and Compliance
Best Practices
Support
Changelog