Note:
Based on a persistent connection, instructions are sent to drive the AI digital human.
Persistent connection instructions are divided into sending text instructions, sending audio instructions, and sending heartbeat instructions.
Sending a text instruction involves sending broadcast text to the cloud, which synthesizes speech to drive the AI digital human to speak;
Sending an audio instruction involves sending voice stream shards to the cloud, which then uses the original sound or voice changer to drive the AI digital human to speak;
Sending a heartbeat instruction is to maintain a persistent connection and prevent session disconnection. If the client does not send a heartbeat and there is no valid data exchanged, the cloud will automatically disconnect the persistent connection after 3 minutes and close the session stream after 10 minutes.
The relationship between the DriverType parameter and the instruction types when creating a new stream is as follows:
DriverType 1: Text-driven; supporting sending text instructions (SEND_TEXT) and heartbeat instructions (SEND_HEARTBEAT).
DriverType 3: Audio-driven; supporting sending text instructions (SEND_TEXT), audio instructions (SEND_AUDIO), and heartbeat instructions (SEND_HEARTBEAT).
1. When a text is being broadcast, if you want to send audio, you need to send an interrupt with an empty text instruction until you receive a TextOver event, after which you can send the audio.
2. When the text is being broadcast and you want to broadcast new text, directly send the new text instruction. The current text broadcast will be interrupted, and upon completion, you will receive a TextOver event. The broadcast of the new text will start with a TextStart event and end with a TextOver event.
3. When an audio-driven session is in progress, if you want to interrupt the audio to send a text instruction, you need to send an audio instruction with IsFinal set to true to end the audio session. Only after receiving an AudioOver event can you send text or audio.