> ## Documentation Index
> Fetch the complete documentation index at: https://docs.modellix.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Modellix Live Overview

> Real-time voice, vision, speech transcription, and speech translation over WebSocket. Connect through Modellix with one API key and pay for the usage each model reports or the audio you send.

Modellix Live is a WebSocket passthrough for real-time models such as Gemini Live and GPT Realtime. Open one connection and stream the input supported by your model: voice and vision models return spoken replies, transcription models return text, and translation models return translated speech and text. Authentication, balance checks, and billing are handled by Modellix; each model keeps the message format of its upstream Live API.

> [!NOTE]
>
> Live is a long-lived WebSocket session. It does not use the async task flow of
> Media Models (`https://api.modellix.ai` task polling) or the HTTP request and
> response model of LLM (`https://llm.modellix.ai`).

## What You Get

> [card] Native Live protocol
>
> Frames are forwarded verbatim in both directions. The gateway rewrites only
> the model identifier: `setup.model` on Gemini models, and `session.model` or
> the transcription model on OpenAI models. Official Live client code works
> with a URL and key change.

> [card] One API key
>
> Authenticate with `Authorization: Bearer` using a Modellix key from the
> [console](https://modellix.ai/console/api-key)—not a vendor credential.

> [card] Usage-based billing
>
> Gemini and GPT Realtime conversation models are billed from upstream token
> usage. GPT Live 1 and OpenAI transcription models use reported seconds;
> translation uses the audio seconds you send.

> [card] Multimodal input
>
> Gemini 3.8 Live accepts text, 16 kHz PCM audio, and JPEG video frames. GPT
> Realtime 2.1 accepts text, audio, and images. Transcription and translation
> models accept audio.

## Base URL and Auth

| Item | Value |
| - | - |
| Endpoint | `wss://api.modellix.ai/api/v1/live/{provider}/{model_id}` |
| Gemini Example | `wss://api.modellix.ai/api/v1/live/google/gemini-3.8-live` |
| OpenAI Example | `wss://api.modellix.ai/api/v1/live/openai/gpt-realtime-2.1` |
| Auth | `Authorization: Bearer <Modellix API Key>` on the WebSocket handshake |

```http theme={null}
Authorization: Bearer mdlx-xxxxxxxx
```

> [!WARNING]
>
> Only the `Authorization` header is accepted. Do not pass the key in the URL
> query string.

Browsers cannot set custom headers on a WebSocket handshake. Connect from your backend, or from a client runtime that supports request headers.

## Supported Models

| Model ID | Input | Output | Context / Session Limit | Parameters |
| - | - | - | - | - |
| `google/gemini-3.8-live` | Text, audio, video frames | Audio (plus text transcription) | 131,072 input tokens, 65,536 output tokens | [Gemini 3.8 Live](https://docs.modellix.ai/live/models/gemini-3-8-live.md) |
| `google/gemini-3.5-transcribe-live` | Audio | Interim and final text transcripts | Up to 10 minutes per session | [Gemini 3.5 Transcribe Live](https://docs.modellix.ai/live/models/gemini-3-5-transcribe-live.md) |
| `openai/gpt-realtime-2.1` | Text, audio, images | Text or audio | 128,000-token context, 32,000 output tokens | [GPT Realtime 2.1](https://docs.modellix.ai/live/models/gpt-realtime-2-1.md) |
| `openai/gpt-realtime-2.1-mini` | Text, audio, images | Text or audio | 128,000-token context, 32,000 output tokens | [GPT Realtime 2.1 Mini](https://docs.modellix.ai/live/models/gpt-realtime-2-1-mini.md) |
| `openai/gpt-live-1` | Audio, text context | Audio plus transcripts | Modellix connection limit | [GPT Live 1](https://docs.modellix.ai/live/models/gpt-live-1.md) |
| `openai/gpt-live-transcribe` | Audio | Partial and final text transcripts | Modellix connection limit | [GPT Live Transcribe](https://docs.modellix.ai/live/models/gpt-live-transcribe.md) |
| `openai/gpt-realtime-whisper` | Audio | Partial and final text transcripts | Modellix connection limit | [GPT Realtime Whisper](https://docs.modellix.ai/live/models/gpt-realtime-whisper.md) |
| `openai/gpt-realtime-translate` | Audio | Translated audio and transcripts | Modellix connection limit, one output language per session | [GPT Realtime Translate](https://docs.modellix.ai/live/models/gpt-realtime-translate.md) |

Each model has its own page with a quick start, request parameters, returned messages, formats, limits, and examples.

## Session Flow

The session flow depends on the model's upstream protocol.

### Gemini Models

**1. Open the Connection**

Connect with the `Authorization` header. If the handshake fails, you get an
HTTP status instead of a WebSocket (see [Handshake Errors](#handshake-errors)).

**2. Send Setup**

The first frame must be a `setup` message and must arrive within 30 seconds.
Configure the options supported by your model. Transcribe Live uses
`responseModalities: ["TEXT"]` and `inputAudioTranscription`.
Wait for `setupComplete` before sending input.

**3. Stream Input**

Send `realtimeInput` messages with the input your model supports.
Replies stream back as `serverContent` frames.

**4. Turn Complete**

Gemini 3.8 Live completes a turn with `serverContent.turnComplete`.
Transcribe Live completes an utterance with `serverContent.generationComplete`;
its final transcript carries `usageMetadata`. Modellix bills the completed
turn or utterance from the upstream usage without changing those frames.

**5. Close**

Close the WebSocket normally (code `1000`) when you are done.

### OpenAI Models

Connect with the `Authorization` header, then follow the flow for your model.

| Model | Start | Input | Finish |
| - | - | - | - |
| GPT Realtime 2.1, GPT Realtime 2.1 Mini | Read `session.created`, then send `session.update` | `input_audio_buffer.append` or `conversation.item.create`; send `response.create` when VAD does not start a response | Each response ends with `response.done` |
| GPT Live 1 | Send `session.start` within 30 seconds, then wait for `session.started` | A continuous `session.input_audio.append` stream | Send `session.close`, then read until `session.closed` |
| GPT Live Transcribe, GPT Realtime Whisper | Read `session.created`, then send `session.update` with `session.type` set to `transcription` | `input_audio_buffer.append`, then `input_audio_buffer.commit` for each turn | Each turn ends with `conversation.item.input_audio_transcription.completed` |
| GPT Realtime Translate | Read `session.created`, then send `session.update` with the output language | `session.input_audio_buffer.append` | Send `session.close`, then read until `session.closed` |

## Billing

* **Gemini: per turn or utterance.** Gemini 3.8 Live uses `turnComplete`; Transcribe Live uses `generationComplete`. Each completion produces a request record, with charges based on the upstream token usage. Interim transcription updates are previews and are not counted as additional output.
* **Gemini: by modality.** Input text, image, video, and audio, and output text, audio, and video, are priced separately. Thinking tokens are billed at the output text rate. Unit prices and invoices follow the console.
* **Gemini: context depends on the model.** Gemini 3.8 Live includes accumulated conversation context in each turn; see [Sessions and Limits](https://docs.modellix.ai/live/models/gemini-3-8-live.md#sessions-and-limits). Transcribe Live reports audio input and text output usage for each finalized utterance.
* **GPT Realtime 2.1 and Mini: per response.** Each response is billed from the token usage in `response.done`. See [GPT Realtime 2.1 Billing](https://docs.modellix.ai/live/models/gpt-realtime-2-1.md#billing).
* **GPT Live 1: by session seconds.** Duration comes from the upstream `usage.seconds` and includes silence and waiting time. A session can produce several request records. See [GPT Live 1 Billing](https://docs.modellix.ai/live/models/gpt-live-1.md#billing).
* **GPT Live Transcribe: by transcribed seconds.** Each completed turn is billed from its reported `usage.seconds`. See [GPT Live Transcribe Billing](https://docs.modellix.ai/live/models/gpt-live-transcribe.md#billing).
* **GPT Realtime Whisper: by input audio seconds.** Each completed turn produces one request record. See [GPT Realtime Whisper Billing](https://docs.modellix.ai/live/models/gpt-realtime-whisper.md#billing).
* **GPT Realtime Translate: by input audio seconds.** Seconds are the decoded audio bytes successfully forwarded, divided by 48,000. Silence you send counts; gaps with nothing sent do not. See [GPT Realtime Translate Billing](https://docs.modellix.ai/live/models/gpt-realtime-translate.md#billing).
* **Incomplete sessions.** If a connection ends in the middle of a turn or response, the usage received so far is billed. Modellix does not estimate missing usage. GPT Realtime Translate bills the audio already forwarded.
* **Balance.** The session starts only with sufficient balance. If the balance runs out mid-session, the connection is closed with code `4402` after the current turn or billing batch at the latest.

## Errors

### Handshake Errors

Failures before the WebSocket upgrade return an HTTP status with the same JSON error body as the Media API:

```json theme={null}
{
  "code": 401,
  "message": "Authentication failed: invalid API key"
}
```

* `code` (integer) equals the HTTP status code.
* `message` (string) is formatted as `"<Category>: <detail>"`.

| HTTP | Description | Retryable |
| - | - | - |
| `400` | Not a WebSocket upgrade request, or invalid parameters | No. Fix the request. |
| `401` | Missing, malformed, or invalid API key | No. Provide a valid key. |
| `402` | Insufficient balance | No. Recharge your account. |
| `404` | Model does not exist or does not support Live | No. Check the model ID. |
| `429` | Rate limit or concurrent session limit reached | Yes. Back off, or close another session. |
| `500` | Internal gateway error | Yes. Retry with backoff. |
| `503` | Service is restarting or at capacity. May include `Retry-After`. | Yes. Retry after the delay. |

### Close Codes

After the connection is open, the server ends a session with a WebSocket close code and a short reason.

| Code | Reason | Meaning | Reconnect |
| - | - | - | - |
| `1000` | | Normal closure | |
| `1001` | `service is restarting` | The service is shutting down. | Yes |
| `1001` | `ping timeout` | The server did not receive a pong in time. | Yes |
| `1008` | `timed out waiting for setup message` | `setup` was not sent within 30 seconds. | After fixing |
| `1008` | `timed out waiting for session.start` | GPT Live 1: `session.start` was not sent within 30 seconds. | After fixing |
| `1008` | `invalid live initialization` | GPT Live 1: the first frame was not a valid `session.start`. | After fixing |
| `1008` | `invalid live frame` | OpenAI models: a gateway-controlled JSON key has invalid casing or is duplicated, or a nested `audio`, `input`, or `delegation` object has the wrong type. | After fixing |
| `1008` | `invalid translation audio` | GPT Realtime Translate: a binary or non-JSON frame, or invalid audio. | After fixing |
| `1008` | `GPT-Live Responses delegation is not supported` | GPT Live 1: `delegation.type` selected `responses`. | After fixing |
| `1008` | `gpt-realtime-whisper is not supported` | OpenAI models: a session of another model selected this transcription model. | After fixing |
| `1008` | `attached input transcription is not supported` | OpenAI models other than GPT Live Transcribe and GPT Realtime Whisper: `audio.input.transcription` was set to a value other than `null`. | After fixing |
| `1008` | `client send rate exceeded` | Upload rate per connection exceeded. | After slowing down |
| `1008` | Other text | Gemini models: the first frame was not a valid `setup`. | After fixing |
| `1009` | `frame too large` | A message exceeded the maximum size (currently 16 MiB). | After fixing |
| `1011` | `upstream unavailable` | The upstream model is unavailable, or an internal error occurred. | Yes |
| `4402` | `insufficient balance` | Balance exhausted during the session. | After recharging |
| `4408` | `max connection duration reached` | Maximum connection duration reached. | Reconnect; use session resumption only when supported by your model. |

Close codes sent by the upstream model are passed through to you.

### Keepalive

The server sends a WebSocket ping every 30 seconds and closes a connection that stops answering. Standard WebSocket libraries reply with a pong automatically. Do not disable it, and keep your event loop responsive while playing audio.

## Limits

* Messages larger than 16 MiB are rejected with `1009`.
* Upload rate is limited per connection. Send video at 1 frame per second and audio in real time to stay far below the limit; exceeding it closes the connection with `1008`.
* Concurrent sessions are limited per team. Exceeding the limit returns `429` on the handshake.
* A session that is idle still counts toward the concurrent limit. Close connections you no longer use.
* After you disconnect from a GPT Realtime conversation, GPT Live Transcribe, or GPT Realtime Whisper session, Modellix waits up to 2 seconds for final usage. When the service restarts or one of these connections reaches its maximum duration, it stops accepting new input and forwards current output for up to 30 seconds while you remain connected.

## Next Steps

> [card] [Gemini 3.8 Live](https://docs.modellix.ai/live/models/gemini-3-8-live.md)
>
> Quick start, request parameters, returned messages, and examples.

> [card] [Gemini 3.5 Transcribe Live](https://docs.modellix.ai/live/models/gemini-3-5-transcribe-live.md)
>
> Stream audio and receive interim and final text transcripts.

> [card] [GPT Realtime 2.1](https://docs.modellix.ai/live/models/gpt-realtime-2-1.md)
>
> Real-time conversations with text, audio, and image input.

> [card] [GPT Realtime 2.1 Mini](https://docs.modellix.ai/live/models/gpt-realtime-2-1-mini.md)
>
> The Mini variant, with the same protocol as GPT Realtime 2.1.

> [card] [GPT Live 1](https://docs.modellix.ai/live/models/gpt-live-1.md)
>
> Continuous voice sessions billed by session seconds.

> [card] [GPT Live Transcribe](https://docs.modellix.ai/live/models/gpt-live-transcribe.md)
>
> Stream audio and receive partial and final text transcripts.

> [card] [GPT Realtime Whisper](https://docs.modellix.ai/live/models/gpt-realtime-whisper.md)
>
> Stream audio and receive transcripts, billed by input audio seconds.

> [card] [GPT Realtime Translate](https://docs.modellix.ai/live/models/gpt-realtime-translate.md)
>
> Stream speech and receive translated audio and text.

> [card] [Get an API Key](https://modellix.ai/console/api-key)
>
> Create a Modellix API key in the console.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.