Skip to content
Documentation
Esc
↑↓navigate↵open⌘Jpreview
On this page

WebSocket & fMP4 media

The media WebSocket — connection, token auth, fMP4 streaming to MSE, message catalog, and the conversational turn lifecycle.

The media WebSocket carries the avatar video stream (fMP4 segments for MSE playback) and the SDK control messages. This guide covers connecting, consuming the media stream, the message catalog, and how a conversational turn flows.

Connecting

The full connection URL (ws_url) — token already embedded — comes from POST /v1/sessions or a client-token refresh:

wss://<orvyn-public-api-host>/v1/sessions/{session_id}/ws?token=...

Upgrade validation, in order: session id format → token configured → token present → token signature/expiry/binding → tenant ownership → rate limit (connection attempts only; an open connection is not rate-limited).

Receiving the media stream

Media is delivered as fragmented MP4 for Media Source Extensions playback: one avatar.fmp4.init segment, then avatar.fmp4.segment frames in order.

const mediaSource = new MediaSource();
videoEl.src = URL.createObjectURL(mediaSource);

mediaSource.addEventListener("sourceopen", () => {
  const sb = mediaSource.addSourceBuffer('video/mp4; codecs="avc1.42E01E"');
  // fmp4_av also needs audio: 'video/mp4; codecs="avc1.42E01E,mp4a.40.2"'

  ws.onmessage = (e) => {
    const msg = JSON.parse(e.data);
    if (msg.type === "avatar.fmp4.init") {
      sb.appendBuffer(base64ToBuffer(msg.dataBase64));
    } else if (msg.type === "avatar.fmp4.segment") {
      sb.appendBuffer(base64ToBuffer(msg.dataBase64));
    }
  };
});

Append the init segment first, then all segments in order — MSE handles playback buffering automatically. Set ws.binaryType = "arraybuffer" before connecting so binary audio frames (microphone input) are received correctly.

Which segment fields do I need?

For playback: dataBase64, sequence, mode ("idle" or "speech"), isSpeechStart, isSpeechEnd, and turnId. Diagnostic fields (render timings, counters) can be ignored — they exist for latency dashboards.

Message catalog

All JSON messages key on type. Binary frames are raw audio (client → server only). Unknown type values and unknown fields may appear at any time — ignore them; the protocol is forward-compatible.

Client → Server

Message Purpose
mic.start Start microphone audio input
mic.stop Stop microphone audio input
session.end End the session over WebSocket
avatar.speech.playback.ended Acknowledge a speech turn finished playing
(binary PCM16 frames) Microphone audio: 16 kHz mono, signed 16-bit LE

Server → Client

Message Purpose
session.ready Avatar loaded, rendering active (includes dimensions, fps, render mode)
session.ended Session fully torn down — close the socket after this
status Pipeline state update (mic_ready, thinking, synthesizing, …)
avatar.fmp4.init fMP4 initialization segment (feed to MSE SourceBuffer)
avatar.fmp4.segment fMP4 media segment (mode: "idle" or "speech")
speech.started User speech turn detected
speech.finalized User speech turn complete — pipeline runs
transcript.final Final transcription for the turn
assistant.text LLM response text
assistant.audio TTS audio (only in fmp4_video mode; muxed into fMP4 in fmp4_av)
error Error message — check code, message, and optional fatal

Sending microphone audio

After mic.start, stream raw audio as binary WebSocket frames:

Property Value
Format PCM16 (signed 16-bit little-endian)
Sample rate 16 000 Hz
Channels 1 (mono)
Frame type ArrayBuffer (binary)

Chunks must be non-empty with an even byte count — malformed chunks return an invalid_audio_chunk error.

The conversational turn lifecycle

A “turn” is one conversational exchange, correlated by a monotonically increasing turnId:

Speech detected

Server emits speech.started — the turn opens.

Turn-final detection

Smart Turn decides when the user finished; while it predicts the user is still speaking the turn stays open (status: "listening_same_turn").

Pipeline runs

On speech.finalized: transcription → response generation → speech synthesis.

Avatar speaks

Speech-mode fMP4 segments stream (isSpeechStart: true on the first). Your client acknowledges with avatar.speech.playback.ended when playback finishes, and idle animation resumes.

Errors on the socket

Socket errors arrive as { "type": "error", "code": "...", "message": "...", "fatal": false }:

  • Fatal errors close the socket — only invalid_session (session missing after a server restart) is fatal today. Create a new session.
  • Lifecycle errors (e.g. avatar_not_found, session_already_active) do not close the socket. Follow the message guidance.
  • Pipeline errors (stt_failed, llm_failed, tts_failed, avatar_render_failed) are per-turn — the session survives and the next turn can succeed. The avatar may not respond for the failed turn.

Render modes

session.ready includes the active renderMode:

Mode Video Audio
fmp4_av fMP4 segments Muxed into the fMP4 stream (default)
fmp4_video fMP4 segments Separate WAV via assistant.audio

Transport selection

The transport field on session creation chooses the media delivery:

Value Behavior
websocket_fmp4 Forces fMP4 over WebSocket. Never falls back.
webrtc_sfu Requests the WebRTC SFU transport. Available only when the platform has SFU enabled for your account; returns an error instead of falling back if unavailable.
auto (default) Server chooses — prefers SFU when configured, otherwise fMP4.

Media transport is replaceable by design: session creation semantics do not change when the transport changes — only stream.transport and the connection fields.

Refreshing the token

Reconnects after the 5-minute token expiry use the canonical refresh endpoint:

curl -X POST \
  "https://<orvyn-public-api-host>/v1/sessions/$SESSION_ID/client-token" \
  -H "Authorization: Bearer $AVTR_API_KEY"

Was this page helpful?