> ## Documentation Index
> Fetch the complete documentation index at: https://docs.modellix.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Gemini 3.8 Live

> Quick start, request parameters, messages, formats, limits, and FAQ for google/gemini-3.8-live: real-time voice and vision sessions over WebSocket.

> [card] WSS
>
> `wss://api.modellix.ai/api/v1/live/google/gemini-3.8-live`

| Item | Value |
| - | - |
| Model ID | `google/gemini-3.8-live` |
| Input | Text, audio (16 kHz PCM), video frames (JPEG) |
| Output | Audio (24 kHz PCM), with optional text transcription |
| Context | 131,072 input tokens, 65,536 output tokens |
| Auth | `Authorization: Bearer <Modellix API Key>` on the handshake |

For the session flow, billing, close codes, and handshake errors shared by all Live models, see the [Live Overview](https://docs.modellix.ai/live/overview.md).

## Quick Start

```python Python theme={null}
# pip install "websockets>=14"
import asyncio
import json
import os

import websockets

URL = "wss://api.modellix.ai/api/v1/live/google/gemini-3.8-live"


async def main() -> None:
    headers = {"Authorization": f"Bearer {os.environ['MODELLIX_API_KEY']}"}
    async with websockets.connect(URL, additional_headers=headers, max_size=16 * 1024 * 1024) as ws:
        # 1. The first frame must be setup.
        await ws.send(json.dumps({
            "setup": {
                "generationConfig": {"responseModalities": ["AUDIO"]},
                "outputAudioTranscription": {},
            }
        }))
        reply = json.loads(await ws.recv())
        assert "setupComplete" in reply, reply

        # 2. Send a text turn.
        await ws.send(json.dumps({"realtimeInput": {"text": "Introduce yourself in one sentence."}}))

        # 3. Read frames until the turn completes.
        async for raw in ws:
            message = json.loads(raw)
            server = message.get("serverContent") or {}
            text = (server.get("outputTranscription") or {}).get("text")
            if text:
                print(text, end="", flush=True)
            if message.get("usageMetadata"):
                print("\nUsage:", message["usageMetadata"])
            if server.get("turnComplete"):
                break


asyncio.run(main())
```

```javascript Node.js theme={null}
// npm install ws
import WebSocket from "ws";

const url = "wss://api.modellix.ai/api/v1/live/google/gemini-3.8-live";
const ws = new WebSocket(url, {
  headers: { Authorization: `Bearer ${process.env.MODELLIX_API_KEY}` },
});

ws.on("open", () => {
  // 1. The first frame must be setup.
  ws.send(JSON.stringify({
    setup: {
      generationConfig: { responseModalities: ["AUDIO"] },
      outputAudioTranscription: {},
    },
  }));
});

ws.on("message", (data) => {
  const message = JSON.parse(data.toString());
  if (message.setupComplete) {
    // 2. Send a text turn.
    ws.send(JSON.stringify({ realtimeInput: { text: "Introduce yourself in one sentence." } }));
    return;
  }
  // 3. Read frames until the turn completes.
  const text = message.serverContent?.outputTranscription?.text;
  if (text) process.stdout.write(text);
  if (message.usageMetadata) console.log("\nUsage:", message.usageMetadata);
  if (message.serverContent?.turnComplete) ws.close();
});

ws.on("close", (code, reason) => console.log(`\nClosed: ${code} ${reason}`));
```

Set `MODELLIX_API_KEY` first. Both examples send one text turn and print the spoken reply as text through output transcription.

## Configuration Examples

Each example is a complete `setup` frame. Pick one and adjust.

> [tab] Audio Only
>
> ```json theme={null}
> {
>   "setup": {
>     "generationConfig": { "responseModalities": ["AUDIO"] }
>   }
> }
> ```

> [tab] Transcription
>
> Recommended. You get the spoken reply as audio and as text.
>
> ```json theme={null}
> {
>   "setup": {
>     "generationConfig": { "responseModalities": ["AUDIO"] },
>     "systemInstruction": {
>       "parts": [{ "text": "You are a friendly assistant. Answer naturally and briefly." }]
>     },
>     "inputAudioTranscription": {},
>     "outputAudioTranscription": {}
>   }
> }
> ```

> [tab] Voice
>
> ```json theme={null}
> {
>   "setup": {
>     "generationConfig": {
>       "responseModalities": ["AUDIO"],
>       "speechConfig": {
>         "voiceConfig": { "prebuiltVoiceConfig": { "voiceName": "Kore" } }
>       }
>     },
>     "systemInstruction": {
>       "parts": [{ "text": "You are a concise voice assistant. Respond only in Japanese." }]
>     },
>     "inputAudioTranscription": {},
>     "outputAudioTranscription": {}
>   }
> }
> ```

> [tab] Long Session
>
> Enable compression and resumption in `setup`.
>
> ```json theme={null}
> {
>   "setup": {
>     "generationConfig": { "responseModalities": ["AUDIO"] },
>     "contextWindowCompression": { "slidingWindow": {} },
>     "sessionResumption": {}
>   }
> }
> ```
>
> After `goAway` or a disconnect, reconnect and pass the latest handle.
>
> ```json theme={null}
> {
>   "setup": {
>     "generationConfig": { "responseModalities": ["AUDIO"] },
>     "contextWindowCompression": { "slidingWindow": {} },
>     "sessionResumption": { "handle": "<latest handle>" }
>   }
> }
> ```

> [tab] Manual VAD
>
> Turn VAD off and mark speech yourself.
>
> ```text theme={null}
> → {"setup": {"generationConfig": {"responseModalities": ["AUDIO"]}, "realtimeInputConfig": {"automaticActivityDetection": {"disabled": true}}}}
> ← {"setupComplete": {}}
> → {"realtimeInput": {"activityStart": {}}}
> → {"realtimeInput": {"audio": {"mimeType": "audio/pcm;rate=16000", "data": "..."}}}
> → {"realtimeInput": {"activityEnd": {}}}
> ← {"serverContent": {"modelTurn": {"parts": [{"inlineData": {"mimeType": "audio/pcm;rate=24000", "data": "..."}}]}}}
> ← {"serverContent": {"generationComplete": true, "turnComplete": true}}
> ```

> [tab] Context
>
> Append earlier turns without asking for a reply, then continue.
>
> ```json theme={null}
> {
>   "clientContent": {
>     "turns": [
>       { "role": "user", "parts": [{ "text": "My name is Alex." }] },
>       { "role": "model", "parts": [{ "text": "Nice to meet you, Alex." }] }
>     ],
>     "turnComplete": false
>   }
> }
> ```

## Sending Audio and Video

| Input | Message | Format |
| - | - | - |
| Text | `realtimeInput.text` | UTF-8 string. Triggers a model reply. |
| Audio in | `realtimeInput.audio` | Raw 16-bit PCM, 16 kHz, little-endian, mono, base64. Send in chunks of about 100 ms. |
| Video in | `realtimeInput.video` | Discrete JPEG frames, base64. At most 1 frame per second. |
| Audio out | `serverContent.modelTurn.parts[].inlineData` | Raw 16-bit PCM, 24 kHz, little-endian, mono, base64. |

> [!TIP]
>
> **Send video as a stream, not a single frame.** A single JPEG followed
> immediately by a question may be ignored by the model, and no image tokens are
> counted. Send several frames at up to 1 frame per second, then ask your
> question. Video frames are reported and billed as image input tokens.

> [tab] Audio Input
>
> Stream a 16 kHz PCM recording in 100 ms chunks, followed by silence so the server detects the end of speech.
>
> ```python theme={null}
> import asyncio
> import base64
> import json
>
> CHUNK_BYTES = 3200  # 100 ms of 16 kHz, 16-bit mono PCM
>
>
> async def stream_audio(ws, pcm: bytes) -> None:
>     pcm += bytes(16000 * 2 * 2)  # 2 seconds of silence
>     for start in range(0, len(pcm), CHUNK_BYTES):
>         chunk = base64.b64encode(pcm[start:start + CHUNK_BYTES]).decode()
>         await ws.send(json.dumps({
>             "realtimeInput": {"audio": {"mimeType": "audio/pcm;rate=16000", "data": chunk}}
>         }))
>         await asyncio.sleep(0.1)  # real-time pace
> ```

> [tab] Video Input
>
> Send several frames at 1 frame per second, then ask the question.
>
> ```python theme={null}
> import asyncio
> import base64
> import json
>
>
> async def ask_about_frames(ws, jpeg_frames: list[bytes], question: str) -> None:
>     for frame in jpeg_frames:
>         await ws.send(json.dumps({
>             "realtimeInput": {"video": {"mimeType": "image/jpeg", "data": base64.b64encode(frame).decode()}}
>         }))
>         await asyncio.sleep(1)
>     await ws.send(json.dumps({"realtimeInput": {"text": question}}))
> ```

## Client Messages

Every frame is a JSON text message with exactly one top-level field. Send `setup` first and wait for `setupComplete`. Parameters are optional unless marked **required**. Where a default is not listed, the upstream default applies.

### setup

First frame. It must arrive within 30 seconds and cannot be changed afterwards. Every field is optional. Other upstream fields, such as `tools`, are forwarded unchanged. See the [Gemini Live API reference](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/models/multimodal-live).

**model** `string`

Ignored. The model is selected by the URL, and Modellix replaces this field.

**generationConfig** `object`

Generation settings.

**properties**

**responseModalities** `string[]`

Output modality. Allowed values: `["AUDIO"]`. Use `outputAudioTranscription` to also receive the reply as text.

**speechConfig.voiceConfig.prebuiltVoiceConfig.voiceName** `string`

Voice for audio output. Allowed values: one of the 30 [voices](#voices), for example `Puck`.

**maxOutputTokens** `integer`

Maximum output tokens per turn. Allowed values: `1` to `65,536`.

**systemInstruction** `object`

Instruction that stays in effect for the whole session. Allowed values: `{"parts": [{"text": "..."}]}`, text parts only. It can only be set here; to add context later, use `clientContent`.

**inputAudioTranscription** `object` (default: off)

Allowed values: `{}`. Enables transcription of the audio you send, returned in `serverContent.inputTranscription`.

**outputAudioTranscription** `object` (default: off)

Allowed values: `{}`. Enables the text of the spoken reply, returned in `serverContent.outputTranscription`.

**realtimeInputConfig.automaticActivityDetection** `object`

Server-side voice activity detection (VAD). While it is on, the user can interrupt the model by speaking.

**properties**

**disabled** `boolean` (default: false)

Allowed values: `true`, `false`. Set to `true` to turn VAD off. You then mark speech yourself with `activityStart` and `activityEnd`.

**prefixPaddingMs** `integer`

Audio, in milliseconds, kept from before speech is detected.

**silenceDurationMs** `integer`

Silence, in milliseconds, that ends a speech turn.

**contextWindowCompression** `object` (default: off)

Compresses the context so a session can run past the audio (15 minutes) and audio plus video (2 minutes) limits, and keeps per-turn cost bounded.

**properties**

**slidingWindow** `object`

Allowed values: `{}`. Enables compression with default behavior.

**triggerTokens** `integer`

Context size, in tokens, that triggers compression.

**sessionResumption** `object` (default: off)

Allowed values: `{}`. Enables `sessionResumptionUpdate` messages.

**properties**

**handle** `string`

Continues an earlier session on a new connection. Allowed values: a handle from a previous `sessionResumptionUpdate`. Handles stay valid for 2 hours after the session ends.

> [!NOTE]
>
> Do not set `generationConfig.thinkingConfig.thinkingLevel`; it is not supported for this model. Sampling parameters (`temperature`, `topP`, `topK`) and penalties (`presencePenalty`, `frequencyPenalty`) are not documented here because recent Gemini models ignore or reject them.

### realtimeInput

Streams user input. Send exactly one of the following fields per message. Real-time input can be sent continuously without interrupting the model.

**text** `string`

Text input. Triggers a model reply. Allowed values: non-empty text.

**audio** `object`

Raw 16-bit PCM, 16 kHz, little-endian, mono. Send in small chunks, for example 100 ms.

**properties**

**mimeType** `string` *required*

Allowed values: `audio/pcm;rate=16000`.

**data** `string` *required*

Base64-encoded PCM bytes.

**video** `object`

One JPEG frame. Send at most 1 frame per second, and keep sending while the user talks about the scene. A single frame followed immediately by a question may be ignored.

**properties**

**mimeType** `string` *required*

Allowed values: `image/jpeg`.

**data** `string` *required*

Base64-encoded JPEG bytes.

**audioStreamEnd** `boolean`

Allowed values: `true`. The audio stream ended, for example the microphone was turned off. Send audio again to resume. Use only while VAD is on.

**activityStart** `object`

Allowed values: `{}`. Marks the start of user speech. Use only while VAD is off.

**activityEnd** `object`

Allowed values: `{}`. Marks the end of user speech. Use only while VAD is off.

### clientContent

Appends turns to the conversation history. Use it to restore earlier context or to send text turns with explicit roles.

**turns** `object[]`

Turns appended in order.

**properties**

**role** `string`

Author of the turn. Allowed values: `user`, `model`.

**parts** `object[]` *required*

Text of the turn. Allowed values: `[{"text": "..."}]`.

**turnComplete** `boolean` (default: false)

Allowed values: `true`, `false`. If `true`, the model starts replying now and interrupts any generation in progress. Otherwise the server waits for more input.

## Server Messages

**setupComplete** `object`

The session is ready for input. Always the first message after `setup`.

**serverContent** `object`

Model output, transcriptions, and turn signals. Output streams across many frames and is generated faster than real time; buffer audio and play it back at 24 kHz.

**properties**

**modelTurn.parts\[].inlineData** `object`

Audio chunk. `mimeType` is `audio/pcm;rate=24000`; `data` is base64 16-bit PCM.

**inputTranscription.text** `string`

Transcription of the audio you sent, when `inputAudioTranscription` is set.

**outputTranscription.text** `string`

Text of the spoken reply, when `outputAudioTranscription` is set. Transcription is independent of `modelTurn`, so their order is not guaranteed.

**generationComplete** `boolean`

The model finished generating. Playback may still be in progress.

**turnComplete** `boolean`

The turn is complete. This ends a billed turn.

**interrupted** `boolean`

The user interrupted the model. Stop playback and discard buffered audio.

**usageMetadata** `object`

Token usage for the turn, sent at the end of each turn. Billing uses it, split by modality. Video frames are reported as `IMAGE` input.

**properties**

**promptTokenCount** `integer`

Input tokens, including the accumulated context.

**candidatesTokenCount** `integer`

Output tokens.

**thoughtsTokenCount** `integer`

Thinking tokens, billed at the output text rate.

**totalTokenCount** `integer`

Total tokens.

**promptTokensDetails** `object[]`

Input tokens by modality: `{"modality": "TEXT" | "IMAGE" | "AUDIO", "tokenCount": 0}`.

**candidatesTokensDetails** `object[]`

Output tokens by modality, in the same shape.

**sessionResumptionUpdate** `object`

Sent periodically when `sessionResumption` is set. Treat the handle as a credential and do not log it.

**properties**

**newHandle** `string`

Pass as `setup.sessionResumption.handle` on a new connection.

**resumable** `boolean`

Whether the session can be resumed now. Only use handles where this is `true`.

**goAway** `object`

Sent about 60 seconds before the connection ends.

**properties**

**timeLeft** `string`

Time before the connection ends, for example `50s`.

Server frames can arrive as text or binary WebSocket messages. Both carry UTF-8 JSON.

### Example Frame Sequence

A text question with a spoken reply and transcription. Audio data is shortened.

```text theme={null}
→ {"setup": {"generationConfig": {"responseModalities": ["AUDIO"]}, "outputAudioTranscription": {}}}
← {"setupComplete": {}}
→ {"realtimeInput": {"text": "What is the capital of France?"}}
← {"serverContent": {"modelTurn": {"parts": [{"inlineData": {"mimeType": "audio/pcm;rate=24000", "data": "..."}}]}}}
← {"serverContent": {"outputTranscription": {"text": "The capital of France is Paris."}}}
← {"serverContent": {"generationComplete": true, "turnComplete": true}}
← {"usageMetadata": {"promptTokenCount": 20, "candidatesTokenCount": 46, "totalTokenCount": 66, "promptTokensDetails": [{"modality": "TEXT", "tokenCount": 20}], "candidatesTokensDetails": [{"modality": "AUDIO", "tokenCount": 46}]}}
```

## Voices

Set `voiceName` to one of these 30 voices:

| Voice | Voice | Voice |
| - | - | - |
| `Zephyr` (bright) | `Puck` (upbeat) | `Charon` (informative) |
| `Kore` (firm) | `Fenrir` (excitable) | `Leda` (youthful) |
| `Orus` (firm) | `Aoede` (breezy) | `Callirrhoe` (easy-going) |
| `Autonoe` (bright) | `Enceladus` (breathy) | `Iapetus` (clear) |
| `Umbriel` (easy-going) | `Algieba` (smooth) | `Despina` (smooth) |
| `Erinome` (clear) | `Algenib` (gravelly) | `Rasalgethi` (informative) |
| `Laomedeia` (upbeat) | `Achernar` (soft) | `Alnilam` (firm) |
| `Schedar` (even) | `Gacrux` (mature) | `Pulcherrima` (forward) |
| `Achird` (friendly) | `Zubenelgenubi` (casual) | `Vindemiatrix` (gentle) |
| `Sadachbia` (lively) | `Sadaltager` (knowledgeable) | `Sulafat` (warm) |

The model switches language with the conversation. To keep replies in one language, say so in `systemInstruction`, for example `"Respond only in Japanese."`

## Sessions and Limits

Upstream limits apply to this model, independent of Modellix:

| Limit | Value |
| - | - |
| Single connection lifetime | About 10 minutes. A `goAway` message arrives about 60 seconds before the end. |
| Audio-only session context | About 15 minutes without compression |
| Audio plus video session context | About 2 minutes without compression |

To go beyond these limits, configure the following in your `setup` message. Modellix forwards them untouched:

* `contextWindowCompression`: keeps the context bounded with a sliding window, which also controls cost.
* `sessionResumption`: the server sends `sessionResumptionUpdate` messages containing a handle. After `goAway` or a disconnect, open a new connection and pass the latest handle in `setup.sessionResumption.handle` to continue with the previous context.

Each new connection is a new Modellix session. Turn numbering and billing records are independent per connection. Modellix also closes any connection at its maximum connection duration with code `4408`.

## FAQ

> [accordion] How do I choose a voice?
>
> Set `setup.generationConfig.speechConfig.voiceConfig.prebuiltVoiceConfig.voiceName` to one of the 30 [voices](#voices). If you omit it, the upstream default voice is used.

> [accordion] How do I get the reply as text?
>
> The model replies in audio only. Add `outputAudioTranscription: {}` to `setup` and read `serverContent.outputTranscription.text`.

> [accordion] How do I get a transcript of what the user said?
>
> Add `inputAudioTranscription: {}` to `setup` and read `serverContent.inputTranscription.text`.

> [accordion] How do I make the model reply in a specific language?
>
> The model follows the conversation language. To fix it, say so in `systemInstruction`, for example `"Respond only in Japanese."`

> [accordion] How do I interrupt the model?
>
> While voice activity detection is on, the user can interrupt by speaking. You can also send a `clientContent` message with `turnComplete` set to `true`. The server sends `serverContent.interrupted`; stop playback and discard buffered audio.

> [accordion] Does it support video?
>
> Yes. Send JPEG frames through `realtimeInput.video` at up to 1 frame per second. See [Sending Audio and Video](#sending-audio-and-video).

> [accordion] How do I keep a session running longer than 10 minutes?
>
> Enable `contextWindowCompression` and `sessionResumption`, then reconnect with the latest handle after `goAway`. See [Sessions and Limits](#sessions-and-limits).

> [accordion] How do I see token usage?
>
> Read `usageMetadata` at the end of each turn. Billing uses it, split by modality.

> [accordion] Can I use tools or function calling?
>
> `setup.tools` and the related messages are forwarded unchanged. See the upstream [Gemini Live API reference](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/models/multimodal-live) for their format.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.