Skip to main content

Best practices for high-concurrency speech synthesis

Reuse synthesis objects and WebSocket connections with object pools, and configure concurrency, warm-up, and error handling to reduce connection overhead.

Scope

Object pools retain synthesis objects and their WebSocket connections to reduce object creation, TCP/TLS handshakes, and protocol upgrades. The following model series and SDKs support object pooling:
  • Qwen-Audio-TTS, CosyVoice: Python and Java object pools.

Prerequisites

  • Get an API key and set DASHSCOPE_API_KEY. Use a key for the region you call.
  • Replace {WorkspaceId} with the region’s workspace ID and configure its WebSocket endpoint.
  • Python SDK
  • Java SDK
Qwen-Audio-TTS and CosyVoice object pooling requires DashScope Python SDK 1.25.2 or later:
pip install "dashscope>=1.25.2"

Configure concurrency and resource pools

Configure resource pools for the expected number of simultaneous tasks and server capacity. Model limits apply to the Alibaba Cloud account and are shared by its RAM users, workspaces, and API keys. For model request-rate and concurrency limits, see Rate limits.
  • Start with a pool capacity of 1.5–2 times peak concurrency and adjust for the actual workload.
  • The Python pool may create extra objects when all pooled objects are borrowed. These objects need new connections when used.
  • Java settings
  • Server configuration reference
Set SDK environment variables before starting the Java process. The first three variables are SDK settings; the pool capacity variables below them are defined by the examples:
Environment variableDefaultDescription
DASHSCOPE_CONNECTION_POOL_SIZE32The underlying OkHttp idle connection pool setting.
DASHSCOPE_MAXIMUM_ASYNC_REQUESTS32The maximum requests setting for the underlying asynchronous dispatcher.
DASHSCOPE_MAXIMUM_ASYNC_REQUESTS_PER_HOST32The per-host maximum requests setting for the underlying asynchronous dispatcher.
COSYVOICE_OBJECTPOOL_SIZE500Object pool capacity in the Qwen-Audio-TTS/CosyVoice Java example, used for maxTotal and maxIdle. This is not a built-in SDK parameter.
The Java examples set maxIdle equal to maxTotal to reduce object recreation after idle objects are destroyed. Close an object’s connection when destroying it. At maxTotal, new borrow requests wait for an available object.

Code examples

Qwen-Audio-TTS/CosyVoice

The examples use qwen-audio-3.1-tts-flash and longanhuan_v3.1 for three concurrent synthesis tasks. For CosyVoice, change the model and voice to cosyvoice-v3-flash and longanyang. For other models and voice requirements, see Real-time speech synthesis.
  • Python SDK
  • Java SDK
import os
import threading
from concurrent.futures import ThreadPoolExecutor
import dashscope
from dashscope.audio.tts_v2 import SpeechSynthesizerObjectPool, ResultCallback, AudioFormat

dashscope.api_key = os.environ["DASHSCOPE_API_KEY"]
dashscope.base_websocket_api_url = "wss://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api-ws/v1/inference"
pool = SpeechSynthesizerObjectPool(max_size=3)

def synthesize(index):
    done = threading.Event()
    errors = []
    with open(f"output_{index}.mp3", "wb") as audio:
        class Callback(ResultCallback):
            def on_data(self, data: bytes):
                audio.write(data)

            def on_complete(self):
                done.set()

            def on_error(self, message):
                errors.append(message)
                done.set()

        synthesizer = pool.borrow_synthesizer(
            model="qwen-audio-3.1-tts-flash",
            voice="longanhuan_v3.1",
            format=AudioFormat.MP3_24000HZ_MONO_256KBPS,
            callback=Callback(),
        )
        successful = False
        try:
            synthesizer.streaming_call("你好，欢迎使用阿里云百炼。")
            synthesizer.streaming_complete(60000)
            if not done.wait(60):
                raise TimeoutError("Synthesis timed out")
            if errors:
                raise RuntimeError(errors[0])
            successful = True
        finally:
            if successful:
                pool.return_synthesizer(synthesizer)
            else:
                synthesizer.close()

try:
    with ThreadPoolExecutor(max_workers=3) as executor:
        list(executor.map(synthesize, range(3)))
finally:
    pool.shutdown()

Production use

Resource management

  1. Configure the API key and endpoint, then create a shared object pool for application tasks.
  2. Borrow and configure an object. In Python, pass the model, voice, format, and callback to borrow_synthesizer. In the Qwen-Audio-TTS/CosyVoice Java example, call borrowObject, then call updateParamAndCallback(param, callback) every time to set the parameters and callback for that task.
  3. Run synthesis. Qwen-Audio-TTS/CosyVoice objects support call or streaming_call in Python, and call or streamingCall in Java.
  4. After normal completion, return the object with return_synthesizer in Python or returnObject in Java. Do not directly return unfinished or failed objects: call the object's close() in Python, or discard it with invalidateObject in Java and let the factory close its connection.
  5. After all tasks finish, release pool resources with shutdown() in Python or the pool's close() in Java.

Authentication and multiple API keys

  • The API key is sent in the Authorization header only during the WebSocket handshake. Later run-task messages do not carry it. Changing a key in parameters or global configuration does not change an existing connection's identity, which can assign identity, quota usage, or charges differently from what you expect.
  • In Python, set dashscope.api_key before creating the pool. Connections established during initialization retain the handshake key, and borrow_synthesizer has no API key parameter. SpeechSynthesizerObjectPool is a process-wide singleton, so isolate different keys in different processes.
  • In Java, create a separate pool for each API key and use the same key throughout each pool. For Qwen-Audio-TTS/CosyVoice, call updateParamAndCallback each time an object is borrowed. This method does not reauthenticate an existing connection.

Warm-up and latency measurement

  1. After startup or network recovery, run warm-up calls and increase concurrency gradually. Before performance testing, warm up at the target concurrency, for example for 1–2 minutes, and collect measurements once connections are being reused steadily.
  2. The Python pool establishes connections during initialization. Java GenericObjectPool creates objects on demand and connects on the first synthesis call, so warming up Java connections requires actual synthesis requests.
  3. The first call includes WebSocket connection overhead. Measure its first-audio latency and total task time separately from calls that reuse a connection. You can also use the real-time factor (RTF, total synthesis time / audio duration) to evaluate synthesis speed.
  4. For Qwen-Audio-TTS, CosyVoice, connections close after 60 seconds of inactivity following task completion. Evaluate the next call after a long idle period as a new connection.
SDK-side first-audio latency includes connection establishment and network transmission, so it is not a measure of model inference time alone. For Qwen-Audio-TTS/CosyVoice, a Python object provides get_first_package_delay() for first-audio latency and get_last_request_id() for the request ID used to correlate request logs. Java objects provide getLastRequestId() for the request ID.

Troubleshooting

TCP connections keep increasing despite steady traffic

Frequently creating synthesis objects without closing old connections can continually increase connection counts and server resource usage. Check that objects are returned after task completion and unused connections are closed. In Java, when maxIdle is less than maxTotal, returned objects above the idle capacity are destroyed. If the factory's destroyObject does not close the WebSocket, the connection may remain open after the object is destroyed. Set maxIdle equal to maxTotal to reduce repeated destruction and creation, and close connections whenever objects are destroyed.

Tasks incur extra waiting time or time out

At maxTotal, borrowing from a Java object pool waits for an available object. Check pool capacity, object returns, and connection cleanup to avoid extra time spent waiting for objects.

Tasks slow down after startup or network recovery

After startup or network recovery, run warm-up calls and increase concurrency gradually to avoid blocking on many simultaneous WebSocket handshakes.

The server returns Invalid action('run-task')! Please follow the protocol!

After a client error interrupts a task, the server may still consider the previous task active. Reusing that connection for a new run-task causes a protocol sequence error. Close the connection and discard the object, then establish a new connection for later tasks.

Calls spike despite steady application traffic

Mass reconnection after startup, network interruptions, or many server errors can cause tasks to accumulate. When connections recover, simultaneous submission of those tasks can create a spike and trigger account limits.
  1. Check for network problems or many server errors before the spike, such as Requests rate limit exceeded, please try again later..
  2. Control concurrency during startup and recovery. In Java, maxTotal limits the number of objects borrowed simultaneously from the same pool.
  3. Check account quotas. To request an increase, follow the procedure in Rate limits.
  4. Adjust server specifications or add servers for the actual workload.

All tasks slow down as concurrency grows

Check whether network bandwidth is saturated or concurrency is too high. Adjust concurrency or server capacity for the workload.