Real-time Avatar Standard

Full request body example:

Basic Field Descriptions

If avatar.id is not provided and only image_uri is submitted, the service will automatically call the "image upload" interface, and return that asset's avatar.id in the response's live.avatar.id.

audio: Audio-to-Text

When enabled, the text content of the user's input audio / the digital human's output audio is returned via WebSocket.

vad: Interrupt Control

Used to control interrupt-related parameters, to better capture the content of a user's turn.

llm: Language Model Output Control

Used to control the output details of the language model, to accommodate different user needs.

idle_timeout_seconds: Session Lifecycle

Used to control the time before automatic disconnection when there is no input, to avoid a session hanging indefinitely with no call activity.

memory_retrieval / knowledge_retrieval: External Memory and Knowledge Base (Beta)

Headers

Path parameters

Response

App WebSocket Control Protocol: /live/ws/live/connect

The App WebSocket is used to initialize the session, send text, interrupt, and hang up. It does not transmit audio or video media.

Query custom voices: GET /live/v1/voices → { "voices": [{ "voice": "my_custom_voice" }] }

Delete a custom voice: DELETE /live/v1/voices/{"my_custom_voice"}

Image Upload API

With this interface you can first upload a local image to Vidu, to use as the digital human image when creating a session. Passing the image in this way as the digital human's image can greatly reduce the time needed to create the digital human. Note: images will be automatically deleted after 90 days.

Supports URL and base64 formats, only 1 image [of a single person] is supported, full-body / half-body, any style. Format: PNG / JPG / JPEG / WEBP; image size ≤ 50MB; base64-decoded byte length < 20M, and must include a content type prefix, e.g. data:image/png;base64,{base64_encode}

Interface Overview

Path parameter avatar_id (string): the digital human persona asset ID. See "Data structure - Asset object" for the returned object fields.

Headers

Path parameters

Response

Deletes a digital human persona asset; content that already references this asset is unaffected. Path parameter avatar_id (string): the digital human persona asset ID.

A successful deletion returns an empty object (HTTP 200).

List Query

GET /live/v1/avatars, pagination parameters are passed via the query string.

Returns: total (integer, total count), items (array, elements are asset objects, see "Data structure - Asset object" for structure).

External Memory API (search_user_memory)

The HTTP JSON protocol that a business-side external memory API must follow.

Calling Convention

memory_retrieval and knowledge_retrieval are both optional fields, corresponding respectively to the digital human's ability to call external memory retrieval and external knowledge retrieval during a conversation. See the [External Memory API] and [External Knowledge API] sections for the protocol details.

Headers

Body parameters

Response

conn_init

When conn_init_ack.success=false, the meaning of error_code and handling recommendations are described consistently in the error response section. This section only retains the request, success response, and failure response examples.

text_msg

The server currently does not return a separate business ack for text_msg; the digital human's reply is output via RTC audio/video, and some text responses may also be delivered via WS.

audio_interrupted

Used for a user to interrupt the current AI output. There is currently no separate ack; the client can handle this based on subsequent audio stopping, changes in the digital human's response, or its own UI state.

call_hangup and force_hangup

In addition to listening for type=6, the client must also listen for WebSocket close / error events as a fallback. After proactively hanging up, it is recommended to first close the WebSocket, then call the RTC SDK's leaveChannel().

5.6.5 session_update

Used to dynamically update some session configuration while a session is in progress (after conn_init_ack success=true), without needing to hang up and rebuild the session.

• Updates are passed in by group: fields not passed in avatar, vad, llm retain their original values;
• knowledge_retrieval / memory_retrieval are replaced as a whole group — passing them in overwrites the entire group's configuration, and sub-fields within the group not passed use default values.
• The server validates the entire update as a whole; if any field is invalid, this update fails entirely, none of the fields take effect, and the original configuration is retained.
• Updates take effect immediately and apply to subsequent conversation turns;
• If the voice or persona is currently playing speech, changes to voice (voice) or persona will be delayed until the current playback ends, to avoid interruption. Updates do not re-validate balance, do not reset session duration, and do not incur additional billing.

Updatable fields and constraints (follows the types and ranges used when creating a session; knowledge_retrieval / memory_retrieval groups are replaced as a whole, other fields only need to pass the parts that need updating):

The update result is returned via session_update_ack:

RTC Join Protocol: AliRTC joinChannel

Why AliRTC appears here: the Live backend is responsible for session creation, authentication, control signaling, and digital human orchestration, but it does not directly forward microphone, camera, or digital human audio/video to the App via WebSocket. Real-time media streams require low-latency transmission, device capture, codec handling, packet loss recovery, and subscription callbacks, so they are carried by the AliRTC SDK.

During integration, two links must be established simultaneously: the App WebSocket handles control, and AliRTC handles media. The two links are parallel and both are required.

AliRTC Prerequisites

• The client needs to integrate the AliRTC SDK separately. For Web, refer to the Alibaba Cloud RTC Web SDK documentation, and load the corresponding SDK on the page.
• The rtc object returned by the create-live interface is the source of join credentials; the client does not need to generate the token itself.
• rtc.token: the authentication information passed to joinChannel.
• rtc.user_id: the current App user's identity in the RTC room, used as the second parameter of joinChannel.
• rtc.channel_id: the RTC room identifier created by the server, mainly used for logging, subscription callbacks, and troubleshooting; the client usually does not need to construct it itself.

Basic Concepts

Join the Channel and Publish Local Media

The client uses the rtc.token and rtc.user_id returned by create-live to join the channel, then publishes the local microphone; in video mode, it also needs to publish the camera.

Subscribe to and Render Digital Human Audio/Video

With default subscription enabled, remote audio is usually automatically subscribed to and played by the SDK; actual playback may still be affected by browser autoplay policies. Remote video needs a <video autoplay playsinline> container prepared, and the remote view should be bound once the subscription state becomes subscribed.

If you cannot see the remote video, first confirm that RTC joined has already appeared in the logs, then confirm whether the digital human's publishing user is online, and whether videoSubscribeStateChanged or screenShareSubscribeStateChanged has entered the subscribed state. If the browser blocks autoplay, guide the user to click play or resume playback after a user gesture.

Role ID Rules

The client generally joins directly using the rtc information returned by the create interface, without needing to construct the token or channel itself. The ID rules above are mainly used for logging, subscription callbacks, and troubleshooting.

Headers

Query parameters

Response

Create a Digital Human Persona Asset

POST /live/v1/avatars, request body is JSON.

Repeatedly submitting the same image will create a new asset record and directly reuse the already-generated description (the new record's status will be success); on first creation, the service will asynchronously generate a description for the image, and processing progress and result are reflected in the returned asset's status field.

Request parameters:

Request example

Response fields

status enum

Headers

Body parameters

Response

• endpoint and authorization come from the memory_retrieval configuration in the CreateLive request.
• The external service should determine the retrieval scope on its own, based on the authentication information, the endpoint binding relationship, or server-side context.
• The request body does not include user_id, character_id, database filter conditions, embedding vectors, or sensitive information toggles.

Request Example

memory_types enum

Response Example

The external API should return JSON. It is recommended to return only the memories array; has_more does not need to be returned.

• memories[].summary: the core content shown to the model, recommended to be a short natural-language sentence, not a large amount of raw chat history.
• memories[].type: memory type, recommended to be consistent with the memory_types enum in the request.
• memories[].confidence: confidence level, recommended range 0..1. High confidence can be used naturally; medium confidence should be used with caution; low confidence should be confirmed with the user rather than treated directly as fact.
• memories[].updated_at: update time, RFC3339 recommended. When old and new memories conflict, prefer the newer memory, but still consider the current user input.
• memories[].source: memory source, e.g. , .

Error Response

When the external API returns a non-2xx status, the system normalizes the tool result into an error output submitted to the model. The system limits the response body size; an overly large response will be treated as response_too_large. It is still recommended to return structured JSON:

Integration Testing Example

The live service has a built-in unauthenticated fake provider, used for local integration testing, demos, and built-in persona memory retrieval. It is not a production user long-term memory service, and does not write to or persist real user memory.

1. HTTP interface

This interface does not require Authorization. If aggregated through a gateway, the path may be:

The fake provider does not validate live_id — even if not provided, it will always return the built-in memory for character 01. It always reads the live.builtin_avatars["01"].memories from the live service configuration, and does not switch characters based on live_id or character_id. Internal rules: read built-in memory → filter by memory_types → simple match-ranking by query/time_hint against summary/keywords → truncate by max_results.

1. Vidu S1 self-call example

Creation validation still requires authorization to be non-empty; the fake provider ignores this header.

1. Demo verification process (character 01)

The current fake persona memory demo always returns perceivable memories for the built-in character character_id="01". It is recommended to also use character 01 when creating a live session, so that the appearance matches the persona memory returned by the fake API. Recommended approach: first call the fake memory API directly to let the user see what "memories" were returned, then ask the digital human the same or a related question in conversation, so the user clearly perceives that the digital human used the data returned by the API.

Recommended demo questions quick reference:

Example (nickname memory), first call the API:

API returns:

The user then asks the digital human "Do you remember what nickname I like you to call me?", and the expected reply naturally includes Ah Che.

When facing the end user, the character should naturally express "I remember..." and should not say "I called an API / tool / fake provider."

External Knowledge API (search_knowledge)

The HTTP JSON protocol that a business-side external knowledge API must follow.

Calling Convention

• endpoint and authorization come from the knowledge_retrieval configuration in the CreateLive request.
• The request body does not include user_id, creator_id, character_id, collection_ids, document_ids, database filter conditions, embedding vectors, permission toggles, or sensitive information toggles.

Request Example

knowledge_types enum

Response Example

The external API should return JSON. It is recommended to return only the knowledge array.

• knowledge[].content: the core content shown to the model, recommended to be a short natural-language sentence or short paragraph, not a large amount of raw document content.
• knowledge[].title / source / url: helps the model judge the source and, when necessary, cite the basis; url is optional.
• knowledge[].type: knowledge type, recommended to be consistent with the knowledge_types enum in the request.
• knowledge[].confidence: confidence level, recommended range 0..1. High confidence can be used naturally; medium confidence should be used with caution; low confidence should be confirmed with the user or stated as not having found clear information.
• knowledge[].updated_at: update time, RFC3339 recommended. When old and new material conflict, prefer the newer material, but still consider the current user input.

Error Response

When the external API returns a non-2xx status, the system normalizes the tool result into an error output submitted to the model. The system limits the response body size; an overly large response will be treated as response_too_large. It is still recommended to return structured JSON:

Integration Testing Example

The live service has a built-in unauthenticated fake provider, used for local integration testing, demos, and external user testing. It is not a production knowledge base service, and does not write to or persist real knowledge content.

1. HTTP interface

This interface does not require Authorization. If aggregated through a gateway, the path may be:

The fake provider does not validate live_id — even if not provided, it will still return built-in demo knowledge. It always reads the built-in demo knowledge from the live service, and does not switch knowledge bases based on live_id or character_id. Internal rules: read built-in candidates → filter by knowledge_types → simple match-ranking by query/time_hint against title/content/keywords → truncate by max_results.

1. Live self-call example

Creation validation still requires authorization to be non-empty; the fake provider ignores this header.

1. Demo verification process

Recommended approach: first call the fake knowledge API directly to let the user see what material the interface returned, then ask the digital human the same or a related question in conversation, so the user clearly perceives that the digital human used the data returned by the API.

First call the API:

API returns:

The user then asks the digital human "How do I claim the firefly badge? What's the password?", and the expected reply naturally uses the booth location B3-17 and the password Blue Whale Seven.

Headers

Path parameters

Response