---
title: WebSocket & fMP4 media
description: The media WebSocket — connection, token auth, fMP4 streaming to MSE, message catalog, and the conversational turn lifecycle.
sidebar:
  label: WebSocket & fMP4
  order: 3
---

The media WebSocket carries the avatar video stream (fMP4 segments for MSE
playback) and the SDK control messages. This guide covers connecting, consuming
the media stream, the message catalog, and how a conversational turn flows.

:::note[Scope]
This is the **SDK/media WebSocket** between your frontend and Orvyn. It is a
transport surface — it is not part of the six-endpoint REST contract, and it is
authenticated with the short-lived WebSocket token, not your API key.
:::

## Connecting

The full connection URL (`ws_url`) — token already embedded — comes from
`POST /v1/sessions` or a
[client-token refresh](#refreshing-the-token):

```
wss://<orvyn-public-api-host>/v1/sessions/{session_id}/ws?token=...
```

:::danger[The URL is a bearer credential]
The `?token=` value authorizes the connection on its own.

- Use the `ws_url` exactly as returned — **do not add auth headers**.
- Never log, screenshot, or store the full URL.
- The token expires in 5 minutes; refresh with the client-token endpoint.
:::

Upgrade validation, in order: session id format → token configured → token
present → token signature/expiry/binding → tenant ownership → rate limit
(connection attempts only; an open connection is not rate-limited).

## Receiving the media stream

Media is delivered as fragmented MP4 for Media Source Extensions playback:
one `avatar.fmp4.init` segment, then `avatar.fmp4.segment` frames in order.

```javascript
const mediaSource = new MediaSource();
videoEl.src = URL.createObjectURL(mediaSource);

mediaSource.addEventListener("sourceopen", () => {
  const sb = mediaSource.addSourceBuffer('video/mp4; codecs="avc1.42E01E"');
  // fmp4_av also needs audio: 'video/mp4; codecs="avc1.42E01E,mp4a.40.2"'

  ws.onmessage = (e) => {
    const msg = JSON.parse(e.data);
    if (msg.type === "avatar.fmp4.init") {
      sb.appendBuffer(base64ToBuffer(msg.dataBase64));
    } else if (msg.type === "avatar.fmp4.segment") {
      sb.appendBuffer(base64ToBuffer(msg.dataBase64));
    }
  };
});
```

Append the init segment first, then all segments in order — MSE handles
playback buffering automatically. Set `ws.binaryType = "arraybuffer"` before
connecting so binary audio frames (microphone input) are received correctly.

<Accordion>
  <AccordionItem title="Which segment fields do I need?">
    For playback: `dataBase64`, `sequence`, `mode` (`"idle"` or `"speech"`),
    `isSpeechStart`, `isSpeechEnd`, and `turnId`. Diagnostic fields (render
    timings, counters) can be ignored — they exist for latency dashboards.
  </AccordionItem>
</Accordion>

## Message catalog

All JSON messages key on `type`. Binary frames are raw audio (client → server
only). Unknown `type` values and unknown fields may appear at any time —
**ignore them**; the protocol is forward-compatible.

### Client → Server

| Message | Purpose |
|---|---|
| `mic.start` | Start microphone audio input |
| `mic.stop` | Stop microphone audio input |
| `session.end` | End the session over WebSocket |
| `avatar.speech.playback.ended` | Acknowledge a speech turn finished playing |
| *(binary PCM16 frames)* | Microphone audio: 16 kHz mono, signed 16-bit LE |

:::note[No session.start in the canonical flow]
Under canonical V0 semantics, `POST /v1/sessions` already started the session —
the server treats a legacy `session.start` control message as a no-op and emits
`session.ready` regardless. New integrations should not send it.
:::

### Server → Client

| Message | Purpose |
|---|---|
| `session.ready` | Avatar loaded, rendering active (includes dimensions, fps, render mode) |
| `session.ended` | Session fully torn down — close the socket after this |
| `status` | Pipeline state update (`mic_ready`, `thinking`, `synthesizing`, …) |
| `avatar.fmp4.init` | fMP4 initialization segment (feed to MSE SourceBuffer) |
| `avatar.fmp4.segment` | fMP4 media segment (`mode: "idle"` or `"speech"`) |
| `speech.started` | User speech turn detected |
| `speech.finalized` | User speech turn complete — pipeline runs |
| `transcript.final` | Final transcription for the turn |
| `assistant.text` | LLM response text |
| `assistant.audio` | TTS audio (only in `fmp4_video` mode; muxed into fMP4 in `fmp4_av`) |
| `error` | Error message — check `code`, `message`, and optional `fatal` |

## Sending microphone audio

After `mic.start`, stream raw audio as binary WebSocket frames:

| Property | Value |
|---|---|
| Format | PCM16 (signed 16-bit little-endian) |
| Sample rate | 16 000 Hz |
| Channels | 1 (mono) |
| Frame type | `ArrayBuffer` (binary) |

Chunks must be non-empty with an even byte count — malformed chunks return an
`invalid_audio_chunk` error.

## The conversational turn lifecycle

A "turn" is one conversational exchange, correlated by a monotonically
increasing `turnId`:

1. **Speech detected**

    Server emits `speech.started` — the turn opens.

2. **Turn-final detection**

    Smart Turn decides when the user finished; while it predicts the user is
    still speaking the turn stays open (`status: "listening_same_turn"`).

3. **Pipeline runs**

    On `speech.finalized`: transcription → response generation → speech
    synthesis.

4. **Avatar speaks**

    Speech-mode fMP4 segments stream (`isSpeechStart: true` on the first). Your
    client acknowledges with `avatar.speech.playback.ended` when playback
    finishes, and idle animation resumes.

## Errors on the socket

Socket errors arrive as `{ "type": "error", "code": "...", "message": "...", "fatal": false }`:

- **Fatal errors** close the socket — only `invalid_session` (session missing
  after a server restart) is fatal today. Create a new session.
- **Lifecycle errors** (e.g. `avatar_not_found`, `session_already_active`) do
  not close the socket. Follow the message guidance.
- **Pipeline errors** (`stt_failed`, `llm_failed`, `tts_failed`,
  `avatar_render_failed`) are per-turn — the session survives and the next turn
  can succeed. The avatar may not respond for the failed turn.

## Render modes

`session.ready` includes the active `renderMode`:

| Mode | Video | Audio |
|---|---|---|
| `fmp4_av` | fMP4 segments | Muxed into the fMP4 stream (default) |
| `fmp4_video` | fMP4 segments | Separate WAV via `assistant.audio` |

## Transport selection

The `transport` field on session creation chooses the media delivery:

| Value | Behavior |
|---|---|
| `websocket_fmp4` | Forces fMP4 over WebSocket. Never falls back. |
| `webrtc_sfu` | Requests the WebRTC SFU transport. Available only when the platform has SFU enabled for your account; returns an error instead of falling back if unavailable. |
| `auto` (default) | Server chooses — prefers SFU when configured, otherwise fMP4. |

Media transport is replaceable by design: session creation semantics do not
change when the transport changes — only `stream.transport` and the connection
fields.

## Refreshing the token

Reconnects after the 5-minute token expiry use the canonical refresh endpoint:

```bash
curl -X POST \
  "https://<orvyn-public-api-host>/v1/sessions/$SESSION_ID/client-token" \
  -H "Authorization: Bearer $AVTR_API_KEY"
```

:::note[Legacy alias]
The currently-deployed Worker also exposes this refresh at
`POST /v1/sessions/{session_id}/ws-token`. That name is a **non-canonical
bridge alias** scheduled for removal — new integrations use `/client-token`.
:::