Qwen-Audio-3.0-Realtime real-time voice conversation HarmonyOS SDK

​

Quick start

1. Obtain and configure an API key.
2. Download the SDK and run the sample code:
   • Download the latest SDK package.
   • Extract the .tar.gz SDK package. Obtain the HAR SDK from entry/libs and add it to your project dependencies. For C++ integration, use native/libs and native/include in the package to obtain the dynamic libraries and header files.
   • Open the project in DevEco Studio. The sample code is in DashQwenAudioChatPage.ets. Replace the API key to try the feature.

​

Call procedure

1. Initialize the SDK.
2. Set parameters for your use case. Use the parameters argument of initialize to set the connection and control parameters, and use setParams to set the voice conversation parameters.
3. Call startDialog to start the conversation.
4. In onNuiAudioStateChanged, start the recording device based on the audio state.
5. Continuously supply recording data in onNuiNeedAudioData, or call updateAudio to actively push recording data.
6. Continuously receive the audio returned by the model in onNuiAssistEventCallback.
7. Listen for events and obtain event information in onNuiEventCallback.
8. Call stopDialog to stop the conversation, and listen for EVENT_TRANSCRIBER_COMPLETE to confirm that it has ended.
9. When the conversation feature is no longer needed, call release to release the SDK resources.

​

Audio device management

​

Recording (AudioCapturer)

• Create: Call audio.createAudioCapturer(capturerOptions) asynchronously. The sample uses 16 kHz, 16-bit, mono audio (SAMPLE_RATE_16000, CHANNEL_1, SAMPLE_FORMAT_S16LE, and ENCODING_TYPE_RAW).
• Audio source (audio.SourceType):
  • SOURCE_TYPE_MIC: Raw microphone audio. Use this source when SDK-internal AEC is enabled, and send data to the SDK by calling updateAudio.
  • SOURCE_TYPE_VOICE_COMMUNICATION: Call audio on which the system has already performed echo cancellation. Use this source when SDK-internal AEC is disabled, and provide audio through onNuiNeedAudioData.
• Data event: Call capturer.on('readData', (buffer: ArrayBuffer) => void) to continuously obtain recorded audio.
• State event: Call capturer.on('stateChange', (state: audio.AudioState) => void). STATE_RUNNING indicates that recording has started, and STATE_STOPPED indicates that it has stopped.
• Control: Call start() to start, stop() to stop, and release() to release the recorder.

import { audio } from '@kit.AudioKit';

const audioStreamInfo: audio.AudioStreamInfo = {
  samplingRate: audio.AudioSamplingRate.SAMPLE_RATE_16000,
  channels: audio.AudioChannel.CHANNEL_1,
  sampleFormat: audio.AudioSampleFormat.SAMPLE_FORMAT_S16LE,
  encodingType: audio.AudioEncodingType.ENCODING_TYPE_RAW
};
const audioCapturerInfo: audio.AudioCapturerInfo = {
  source: audio.SourceType.SOURCE_TYPE_MIC,
  capturerFlags: 0
};
const options: audio.AudioCapturerOptions = { streamInfo: audioStreamInfo, capturerInfo: audioCapturerInfo };

audio.createAudioCapturer(options).then((capturer) => {
  capturer.on('readData', (buffer: ArrayBuffer) => {
    // Send the recorded audio to the SDK.
    nuiInstance.updateAudio(buffer, false);
  });
  capturer.start();
});

Note: HarmonyOS creates AudioCapturer asynchronously. Call start() only after creation is complete. Do not create and immediately start a recorder in STATE_OPEN. Create it first, then call start() from the STATE_OPEN callback. The sample creates the recorder during doInit and starts it during onNuiAudioStateChanged.

​

Playback (AudioRenderer)

• Create: Call audio.createAudioRenderer(rendererOptions) asynchronously.
• Sample rate: The synthesized response audio returned by DashScope Realtime is 24 kHz. Pass this rate to the AudioPlayer constructor.
• Data event: Call renderer.on('writeData', (data: ArrayBuffer): audio.AudioDataCallbackResult => ...) to provide audio for playback. Return AudioDataCallbackResult.VALID after filling the buffer or INVALID when no data is available.
• State event: Call renderer.on('stateChange', (state: audio.AudioState) => void) to listen for playback start and end events.
• Control: Call start() to start, stop() to stop, and pause() to pause playback. Pausing retains buffered data.

import { audio } from '@kit.AudioKit';

const audioStreamInfo: audio.AudioStreamInfo = {
  samplingRate: audio.AudioSamplingRate.SAMPLE_RATE_24000,
  channels: audio.AudioChannel.CHANNEL_1,
  sampleFormat: audio.AudioSampleFormat.SAMPLE_FORMAT_S16LE,
  encodingType: audio.AudioEncodingType.ENCODING_TYPE_RAW
};
const audioRendererInfo: audio.AudioRendererInfo = {
  usage: audio.StreamUsage.STREAM_USAGE_VOICE_ASSISTANT,
  rendererFlags: 0
};
const options: audio.AudioRendererOptions = { streamInfo: audioStreamInfo, rendererInfo: audioRendererInfo };

audio.createAudioRenderer(options, (err, renderer) => {
  renderer.on('writeData', (data: ArrayBuffer): audio.AudioDataCallbackResult => {
    // Fill data with model response audio from the queue and return VALID or INVALID.
    return audio.AudioDataCallbackResult.VALID;
  });
  renderer.start();
});

AEC reference signal: When SDK-internal AEC is enabled, send the player's output audio to the SDK as the reference signal by calling nuiInstance.pushReferenceData(data, false), which corresponds to updateRefAudio on Android.

​

Permission declaration

{
  "requestPermissions": [
    { "name": "ohos.permission.MICROPHONE" }
  ]
}

​

Request parameters

​

Connection and control parameters

{
  "url": "wss://dashscope.aliyuncs.com/api-ws/v1/inference",
  "apikey": "st-****",
  "device_id": "my_device_id",
  "service_mode": "1"
}

​

Voice conversation parameters

{
  "service_type": 4,
  "nls_config": {
    "model": "qwen-audio-3.1-realtime-plus",
    "sr_format": "pcm"
  }
}

[
  {
    "type": "function",
    "function": {
      "name": "get_weather",
      "description": "Queries weather information for a specified city.",
      "parameters": {
        "type": "object",
        "properties": {
          "city": {
            "type": "string",
            "description": "City"
          }
        },
        "required": [
          "city"
        ]
      }
    }
  }
]

​

Key APIs

​

NativeNui

​

initialize

public initialize(
  callback: INativeNuiCallback,
  parameters: string,
  level: number,
  save_log: boolean = false
): number

​

setParams

public setParams(params: string): number

​

startDialog

public startDialog(vad_mode: Constants.VadMode, dialog_params: string): number

{
  "apikey": "st-****"
}

​

stopDialog

public stopDialog(): number

​

cancelDialog

public cancelDialog(): number

​

dialogAction

public dialogAction(params: string): number

{
  "type": "action",
  "command": "function_call",
  "context": {
    "item": {
      "call_id": "call_xxxx",
      "output": "{\"city\":\"Hangzhou\",\"condition\":\"sunny\",\"temperature\":18}",
      "type": "function_call_output"
    },
    "type": "conversation.item.create"
  }
}

{
  "type": "action",
  "command": "function_call",
  "context": {
    "response": {
      "modalities": ["text", "audio"]
    },
    "type": "response.create"
  }
}

​

updateAudio

public updateAudio(data: ArrayBuffer, first_pack: boolean): number

​

pushReferenceData (corresponds to Android updateRefAudio)

public pushReferenceData(data: ArrayBuffer, first_pack: boolean): number

​

release

public release(): number

​

GetVersion

public GetVersion(): string

​

INativeNuiCallback: listener callbacks

​

onNuiEventCallback: listen for event information

onNuiEventCallback: (
  event: Constants.NuiEvent,
  resultCode: number,
  arg2: number,
  kwsResult: KwsResult,
  asrResult: AsrResult
) => void;

​

onNuiAudioStateChanged: listen for the audio state

onNuiAudioStateChanged: (state: Constants.AudioState) => void

​

onNuiAudioRMSChanged: listen for the recording volume

onNuiAudioRMSChanged: (val: number) => number

​

onNuiNeedAudioData: supply audio data

onNuiNeedAudioData: (buffer: ArrayBuffer) => number

​

onNuiAssistEventCallback: receive auxiliary data and information

onNuiAssistEventCallback?: (
  event: Constants.NuiEvent,
  info: string,
  infoLen: number,
  data: ArrayBuffer
) => void;

Note: This callback is optional (?) on HarmonyOS. Omit it if you do not need the information.

​

onNuiLogTrackCallback: listen for trace logs

onNuiLogTrackCallback: (level: Constants.LogLevel, log: string) => void

​

NuiEvent: event types