Skip to main content
Alibaba Cloud Model Studio home pageAlibaba Cloud Model Studio
Functions Reference

Common audio features

The AOQ Client SDK provides comprehensive audio capabilities, covering audio capture, playback, codec configuration, speaker management, file mixing, external audio stream injection, and audio frame data callbacks. This document introduces common audio features across Android (Java), iOS (Objective-C), and HarmonyOS (ArkTS).

The AOQ Client SDK provides comprehensive audio capabilities, covering audio capture, playback, codec configuration, speaker management, file mixing, external audio stream injection, and audio frame data callbacks. This document introduces common audio features across Android (Java), iOS (Objective-C), and HarmonyOS (ArkTS).

Audio capture

Audio capture opens the device microphone and feeds real-time audio data into the SDK encoding pipeline. The SDK supports two capture modes:
  • Internal capture (default): The SDK automatically manages the microphone — opening, recording, and closing it.
  • External capture: The application manages the microphone directly and feeds the captured PCM data into the SDK through the external audio stream API.

Configuration parameters

Parameter

Type

Default

Description

isExternal

bool

false

Whether to use external capture mode

isVoipMode

bool

false

Whether to enable VoIP mode (hardware AEC). Valid on mobile. If both capture and playback are configured, whichever is set first takes effect.

channel

int

1

Number of capture channels. Supports 1 (mono) or 2 (stereo).

API reference

Function

Android

iOS

HarmonyOS

Start capture

startAudioCapture(config)

startAudioCapture:config:

startAudioCapture(config)

Stop capture

stopAudioCapture()

stopAudioCapture

stopAudioCapture()

Mute/unmute

muteAudioCapture(mute)

muteAudioCapture:

muteAudioCapture(mute)

Example

Android 
AoqAudioCaptureConfig config = new AoqAudioCaptureConfig();
config.isVoipMode = true;
config.channel = 1;
engine.startAudioCapture(config);
iOS 
AoqAudioCaptureConfig *config = [[AoqAudioCaptureConfig alloc] init];
config.isVoipMode = YES;
config.channel = 1;
[engine startAudioCapture:config];
HarmonyOS 
const config: AoqAudioCaptureConfig = { isVoipMode: true, channel: 1 };
engine.startAudioCapture(config);

Audio playback

Audio playback renders received remote audio data to the local speaker or headset. The SDK supports advanced controls including pause/resume with fade in/out and interrupting the current turn of an audio conversation.

Configuration parameters

Parameter

Type

Default

Description

isVoipMode

bool

false

Whether to enable VoIP mode (hardware AEC). Valid on mobile. If both capture and playback are configured, whichever is set first takes effect.

isDefaultSpeaker

bool

true

Whether to use the speaker by default. Valid on mobile and only in non-VoIP mode.

isExternal

bool

false

Whether to use external playback mode.

channel

int

1

Number of playback channels. Supports 1 (mono) or 2 (stereo).

API reference

Function

Android

iOS

HarmonyOS

Start playback

startAudioPlayer(config)

startAudioPlayer:config:

startAudioPlayer(config)

Stop playback

stopAudioPlayer()

stopAudioPlayer

stopAudioPlayer()

Pause playback

pauseAudioPlayer(fadeMs)

pauseAudioPlayer:

pauseAudioPlayer(fadeMs)

Resume playback

resumeAudioPlayer(fadeMs)

resumeAudioPlayer:

resumeAudioPlayer(fadeMs)

Interrupt conversation

interruptAudioPlayer(trackType, fadeMs)

interruptAudioPlayer:fadeMs:

interruptAudioPlayer(trackType, fadeMs)

fadeMs parameter: The fade-in or fade-out duration in milliseconds when pausing or resuming playback. Set to 0 for an immediate switch.

Speaker management

Switch the audio output device between the speaker and earpiece.

Function

Android

iOS

HarmonyOS

Switch speaker

enableSpeakerphone(enable)

enableSpeakerphone:

enableSpeakerphone(enable)

Query speaker state

isSpeakerphoneEnabled()

isSpeakerphoneEnabled

isSpeakerphoneEnabled()

Speaker switching is only allowed in VoIP mode. When not in VoIP mode, calling enableSpeakerphone triggers an OnError(AoqECAudioDeviceEarpieceRequiresVoipMode) error notification.
iOS-specific behavior: iPad devices have only speaker mode. When the AVAudioSession category is not PlayAndRecord, this method always returns YES.

Audio codec configuration

Configure the encoding format, sample rate, channel count, and bitrate for audio uplink (encoder) and downlink (decoder). These settings determine the format for publishing and pulling streams.

Configuration parameters

Parameter

Type

Default

Description

trackType

AoqTrackType

Audio

Audio track type. Currently only one audio stream is supported.

codecType

AoqEncoderType

AudioPCM

Encoding type: AudioPCM(1) or AudioOpus(2)

sampleRate

int

48000

Sample rate. Opus supports 8K/16K/48K. PCM supports 8K/16K/32K/48K.

channel

int

1

Channel count. Supports 1 (mono) or 2 (stereo).

bitrate

int

32000

Bitrate in bps.

API reference

Function

Android

iOS

HarmonyOS

Set encoder config

setAudioEncoderConfig(config)

setAudioEncoderConfig:

setAudioEncoderConfig(config)

Set decoder config

setAudioDecoderConfig(config)

setAudioDecoderConfig:

setAudioDecoderConfig(config)

Supported encoding formats

Enum value

Number

Description

AoqEncoderTypeAudioPCM

1

Raw PCM audio

AoqEncoderTypeAudioOpus

2

Opus encoding

Audio file mixing

Mix a local audio file into the current audio stream for publishing and/or local playback. Each audio file is identified by an application-assigned fileId, allowing multiple file instances to be managed simultaneously.

Mixing configuration parameters

Parameter

Type

Default

Description

fileName

String

-

Audio file path (including filename)

cycles

int

-1

Number of loops. -1 means loop indefinitely.

startPosMs

long

0

Start playback position in milliseconds

publishVolume

int

100

Publishing volume [0–100]

playoutVolume

int

100

Local playback volume [0–100]

API reference

Function

Android

iOS

HarmonyOS

Start playback

startAudioFile(fileId, config)

startAudioFile:config:

startAudioFile(fileId, config)

Stop playback

stopAudioFile(fileId)

stopAudioFile:

stopAudioFile(fileId)

Pause

pauseAudioFile(fileId)

pauseAudioFile:

pauseAudioFile(fileId)

Resume

resumeAudioFile(fileId)

resumeAudioFile:

resumeAudioFile(fileId)

Get file duration

getAudioFileDuration(fileId)

getAudioFileDuration:

getAudioFileDuration(fileId)

Get current position

getAudioFileCurrentPosition(fileId)

getAudioFileCurrentPosition:

getAudioFileCurrentPosition(fileId)

Seek to position

setAudioFilePositionMillis(fileId, pos)

setAudioFilePositionMillis:positionMillis:

setAudioFilePositionMillis(fileId, pos)

Set volume

setAudioFileVolume(fileId, type, vol)

setAudioFileVolume:type:volume:

setAudioFileVolume(fileId, type, vol)

Get volume

getAudioFileVolume(fileId, type)

getAudioFileVolume:type:

getAudioFileVolume(fileId, type)

Volume direction (type): AoqAudioStreamPublish(0) controls the publishing volume. AoqAudioStreamPlayout(1) controls the local playback volume.

State callbacks

State code

Value

Description

AoqAudioFileNone

0

Initial state

AoqAudioFileStarted

1

Playback started

AoqAudioFileStopped

2

Playback stopped

AoqAudioFilePaused

3

Playback paused

AoqAudioFileResumed

4

Playback resumed

AoqAudioFileEnded

5

Playback ended

AoqAudioFileBuffering

6

Buffering

AoqAudioFileBufferingEnd

7

Buffering ended

AoqAudioFileFailed

8

Playback failed

External audio streams

External audio streams let you inject application-generated PCM audio data into the SDK's audio pipeline for publishing and/or local playback. Typical use cases include TTS synthesis output, AI model audio output, and background sound effects. Each external audio stream is identified by an application-assigned streamId.

Configuration parameters

Parameter

Type

Default

Description

trackType

AoqTrackType

Audio

Audio track type

codecType

AoqEncoderType

AudioPCM

Audio stream format

channels

int

1

Channel count

sampleRate

int

48000

Sample rate. Supports 8/12/16/24/32/44.1/48/64/88.2/96/176.4/192 kHz.

playoutVolume

int

100

Local playback volume [0–100]

publishVolume

int

100

Publishing volume [0–100]

maxBufferDuration

int

600000

Maximum buffer duration in milliseconds. Valid range: [100, ~]. Push fails if the buffer is full.

enable3A

bool

false

Whether to apply 3A processing to the input PCM

API reference

Function

Android

iOS

HarmonyOS

Add external stream

addAudioExternalStream(streamId, config)

addAudioExternalStream:config:

addAudioExternalStream(streamId, config)

Push audio data

pushAudioExternalStreamData(streamId, data)

pushAudioExternalStreamData:data:

pushAudioExternalStreamData(streamId, data)

Set volume

setAudioExternalStreamVolume(streamId, type, vol)

setAudioExternalStreamVolume:type:volume:

setAudioExternalStreamVolume(streamId, type, vol)

Get volume

getAudioExternalStreamVolume(streamId, type)

getAudioExternalStreamVolume:type:

getAudioExternalStreamVolume(streamId, type)

Clear buffer

clearAudioExternalStreamBuffer(streamId, fadeoutMs)

clearAudioExternalStreamBuffer:fadeoutMs:

clearAudioExternalStreamBuffer(streamId, fadeoutMs)

Remove stream

removeAudioExternalStream(streamId)

removeAudioExternalStream:

removeAudioExternalStream(streamId)

Best practices for pushing data

  • Call pushAudioExternalStreamData in a loop to ensure data is pushed successfully.
  • If error code 110 (buffer full) is returned, sleep for 30 ms and retry. Do not discard data.
  • Before the engine exits, stop the push loop first, then call removeAudioExternalStream.
  • For real-time capture, each frame is 10 ms — call push whenever data is available. For file-based input, each frame is 40 ms — call push once every 30 ms.

Audio frame callbacks

Audio frame callbacks let you obtain raw PCM data at different points in the audio pipeline, for use in audio analysis, custom processing, recording, and similar scenarios.

Supported data source positions

Data source

Enum value

Description

Captured

0

Raw audio data after capture, before 3A processing

ProcessCaptured

1

Audio data after 3A processing. Callbacks start only after a successful connection.

Publish

2

Audio data about to be published. Requires a successful connection.

Playback

3

Audio data about to be played back (remote downlink)

Callback configuration parameters

Parameter

Type

Default

Description

sampleRate

int

48000

Sample rate for callback audio

channels

int

1

Channel count for callback audio. Supports 1 or 2.

mode

AoqAudioObserverMode

ReadOnly

Read-only (0) or read-write (1) mode

Usage steps

  1. Register the observer: Call setAudioFrameObserver to set the audio frame callback listener.
  2. Enable the data source: Call enableAudioFrameObserver to select the data source position and start callbacks.
  3. Handle callback data: Process PCM data in the callback.

API reference

Function

Android

iOS

HarmonyOS

Register observer

setAudioFrameObserver(listener)

setAudioFrameObserver:

setAudioFrameObserver(observer)

Enable callbacks

enableAudioFrameObserver(enabled, source, config)

enableAudioFrameObserver:audioSource:config:

enableAudioFrameObserver(enabled, source, config)

Callback methods

Callback

Android

iOS

HarmonyOS

Captured data

onCapturedAudioFrame(frame)

onCapturedAudioFrame:

onCapturedAudioFrame(frame)

Post-3A data

onProcessCapturedAudioFrame(frame)

onProcessCapturedAudioFrame:

onProcessCapturedAudioFrame(frame)

Publish data

onPublishAudioFrame(trackType, frame)

onPublishAudioFrame:frame:

onPublishAudioFrame(trackType, frame)

Playback data

onPlaybackAudioFrame(frame)

onPlaybackAudioFrame:

onPlaybackAudioFrame(frame)

Audio state and routing

The SDK automatically monitors audio device state changes and routing switches, and notifies the application layer through callbacks.

Device state codes

State code

Value

Description

AoqAudioDeviceNone

0

Initial state

RecordStarting

1

Capture starting

RecordStarted

2

Capture started

RecordStopping

3

Capture stopping

RecordStopped

4

Capture stopped

RecordFail

5

Capture failed

PlayStarting

6

Playback starting

PlayStarted

7

Playback started

PlayStopping

8

Playback stopping

PlayStopped

9

Playback stopped

PlayFail

10

Playback failed

Device routing types

Route

Value

Description

Default

0

Default

Headset

1

Wired headset

Earpiece

2

Earpiece

HeadsetNoMic

3

Headset without microphone

SpeakerPhone

4

Speaker

Usb

5

USB device

Bluetooth

6

Bluetooth SCO

BluetoothA2dp

7

Bluetooth A2DP

Callback reference

Callback

Android

iOS

HarmonyOS

Device state change

onAudioDeviceStateChanged(state)

onAudioDeviceStateChanged:

onAudioDeviceStateChanged(state, reason)

Route change

onAudioDeviceRouteChanged(routeType)

onAudioDeviceRouteChanged:

onAudioDeviceRouteChanged(routeType)

Device interrupted

onAudioDeviceInterrupted(interrupt)

onAudioDeviceInterrupted:

onAudioDeviceInterrupted(interrupt)

File state

onAudioFileState(state)

onAudioFileState:

onAudioFileState(fileId, stateCode, errorCode)

Audio error and warning codes

Audio error codes

Error code

Value

Description

AoqErrorCodeAudio

100

General audio error

AudioExternalBufferFull

110

External buffer full

AudioDevice

120

General device error

RecordingAuthFailed

121

Microphone permission denied

RecordingOccupied

122

Microphone in use by another process

RecordingBackgroundStart

123

Started recording in the background

RecordingStartFail

124

Failed to start recording

PlayoutOccupied

125

Playback device in use by another process

PlayoutBackgroundStart

126

Started playback in the background

PlayoutStartFail

127

Failed to start playback

EarpieceRequiresVoipMode

128

Earpiece requires VoIP mode to be enabled

Audio warning codes

Warning code

Value

Description

AoqWCAudio

100

General audio warning

AudioHowling

101

Howling detected

AudioDevice

120

General device warning

MicEnumerateError

121

Microphone enumeration error

MicStartTimeout

122

Microphone start timeout

RecordingError

123

Recording error

SpeakerEnumerateError

124

Speaker enumeration error

SpeakerStartTimeout

125

Speaker start timeout

PlayoutError

126

Playback error

iOS only: AVAudioSession control

On iOS, the setAudioSessionRestriction API gives you fine-grained control over how the SDK manages the system AVAudioSession.

Control

Description

SetCategory

Whether the SDK can set the session category

ConfigureSession

Whether the SDK can configure session parameters

DeactivateSession

Whether the SDK can deactivate the session

ActivateSession

Whether the SDK can activate the session

Pass a bitwise combination of restriction values to limit the SDK's control over AVAudioSession and prevent conflicts with other audio components in the application layer.
Alibaba Cloud Model Studio home pageAlibaba Cloud Model Studio
API Usage
Text Generation
Image Generation
  • FAQ
Video Generation
Audio
Real-time Multimodal
Realtime API
Text Embedding
More Models
Toolkit/Framework
Model Production
File Management
More