Realtime Audio

Realtime Audio lets a model listen and speak on the same connection. Your app sends small pieces of microphone audio. The model sends small pieces of its spoken reply back. You can play the reply before it has finished generating.

A WebSocket is just a connection that stays open, so both sides can send messages at any time. You do not upload a new file or make a new HTTP request for each turn.

​

Which voice API do I need?

The last option is the older STT → LLM → TTS pipeline. It also uses a WebSocket, but it is not the same API. Its setup fields and response events are different. You do not need separate STT and TTS models for a native realtime conversation.

​

A conversation in six steps

Your app                         Backboard + the audio model
   |  1. Open a WebSocket                   |
   |  2. Send model and settings --------->|
   |<--------------- 3. session.begin      |
   |  4. Send microphone audio ----------->|
   |<--------------- 5. audio.delta        |  Play each chunk
   |<---------------    transcript.final   |  Show the words
   |<---------------    response.done      |  One reply ended
   |  Send the next turn ----------------->|  Same connection
   |  6. stop ---------------------------->|
   |<---------------    session.ended      |

Transcripts, audio, and tool events can overlap. Do not wait for a transcript before playing audio. response.done means one model response ended, not that the conversation is over. A tool call can be followed by another response.

​

1. Start without setup calls

You do not need to create an assistant or thread first. Realtime follows the same conversation rules as the primary POST /threads/messages endpoint:

An assistant holds reusable configuration. A thread holds one conversation. The ready event returns both IDs. Keep the thread ID to continue, keep only the assistant ID to start fresh with shared configuration, or ignore both for a new conversation next time. Leaving IDs out does not disable saving.

Set BACKBOARD_API_KEY in your backend environment. Use HTTP /threads/messages for text and TTS/STT requests, and the WebSocket below for two-way native audio. The transport is different; the ID lifecycle is the same.

​

2. Choose a model and connect

Get available models rather than guessing a model name:

curl --fail-with-body 'https://app.backboard.io/api/models?model_type=realtime' \
  -H "X-API-Key: $BACKBOARD_API_KEY"

Choose the model’s provider and name. Check its realtime_capabilities for formats and provider settings. Then open:

wss://app.backboard.io/api/threads/realtime

Backend clients authenticate with the X-API-Key WebSocket header. The first message is JSON, not audio:

{
  "provider": "openai",
  "model": "gpt-realtime-mini",
  "system_prompt": "Use simple words. Keep spoken replies short.",
  "provider_options": {
    "audio": {"output": {"voice": "marin"}}
  },
  "memory": "off"
}

Backboard replies with session.begin. This example shows the fields needed for audio; the real event also includes session metadata:

{
  "type": "session.begin",
  "session_id": "SESSION_ID",
  "thread_id": "CREATED_THREAD_ID",
  "assistant_id": "CREATED_ASSISTANT_ID",
  "provider": "openai",
  "model": "gpt-realtime-mini",
  "input_format": {"encoding": "pcm16", "sample_rate": 24000, "channels": 1},
  "output_format": {"encoding": "pcm16", "sample_rate": 24000, "channels": 1}
}

Wait for this event before sending audio. Provider setup events may arrive first; keep reading until session.begin. The SDK does that for you.

To reconnect after ending this session, include the returned thread ID:

{"thread_id":"CREATED_THREAD_ID","provider":"openai","model":"gpt-realtime-mini"}

For a new conversation with the same assistant, include only the assistant ID:

{"assistant_id":"CREATED_ASSISTANT_ID","provider":"openai","model":"gpt-realtime-mini"}

system_prompt and tools are session-only overrides, like per-call settings on the primary messages endpoint. Re-pass them when reconnecting if needed. They do not change the saved assistant. Native provider_options can override instructions again; use system_prompt when you want the same spelling across providers.

​

3. Send the user’s voice

Send each chunk as a binary WebSocket message containing raw audio bytes. For the defaults, this means PCM16: signed 16-bit little-endian samples, one channel. It is not an MP3, WebM, or WAV file, and it has no file header.

Always use the formats in session.begin. At 24 kHz, a 20 ms input chunk is 480 samples, or 960 bytes. At 16 kHz it is 640 bytes. Your microphone may capture at 48 kHz; convert it to the requested rate before sending.

For a recording, convert it first:

ffmpeg -i question.wav -f s16le -acodec pcm_s16le -ac 1 -ar 24000 question.pcm

​

When is the user done speaking?

Automatic turns are the default. Keep sending microphone audio, including silence. The provider detects a pause and starts a reply. Do not send commit after each chunk.

Manual turns are useful for a recording or a push-to-talk button. Disable automatic detection in your setup, send the audio, then send one JSON message:

{"type":"commit"}

Use the following object as provider_options for manual turns:

{"audio":{"input":{"turn_detection":null}}}

​

Start with text instead

You can test speech output without a microphone. After session.begin, send:

{"type":"text","text":"Say hello in one sentence."}

The model answers with the same audio events. No commit is needed for text. You can use text and audio turns on the same session.

​

4. Receive and play the reply

Keep reading the WebSocket while sending audio. Each response event is a JSON text message. Audio bytes are encoded as base64 in data.

These are example events, not a guaranteed order. IDs and text will vary:

{"type":"transcript.final","role":"user","item_id":"user-1","text":"Tell me a fun fact."}

{"type":"audio.delta","data":"BASE64_AUDIO_BYTES","encoding":"pcm16","sample_rate":24000,"channels":1}

{"type":"transcript.delta","role":"assistant","item_id":"reply-1","text":"An octopus has"}

{"type":"transcript.final","role":"assistant","item_id":"reply-1","text":"An octopus has three hearts."}

{"type":"response.done"}

BASE64_AUDIO_BYTES is a placeholder, not playable audio. Decode each real audio.delta.data and queue it in your audio player, in arrival order. Do not treat each chunk as a separate MP3 or call a WAV decoder on raw PCM.

​

A complete audio-in, audio-out API example

This backend Python example uses the raw WebSocket API, not the SDK. It sends question.pcm from the conversion command above and saves the streamed reply as reply.pcm. It uses manual turns so the recording has a clear end.

Use Python 3.11+, install websockets (pip install 'websockets>=14'), and set BACKBOARD_API_KEY. This example automatically creates a fresh conversation; the tool example below adds the tool round trip.

import asyncio
import base64
import json
import os
from pathlib import Path
import websockets

async def main():
    pcm = Path("question.pcm").read_bytes()
    if len(pcm) < 4800 or len(pcm) % 2:
        raise ValueError("Use at least 100 ms of mono PCM16 at 24 kHz")
    async with websockets.connect(
        "wss://app.backboard.io/api/threads/realtime",
        additional_headers={"X-API-Key": os.environ["BACKBOARD_API_KEY"]},
    ) as ws:
        await ws.send(json.dumps({
            "provider": "openai",
            "model": "gpt-realtime-mini",
            "provider_options": {"audio": {"input": {"turn_detection": None}}},
        }))
        async with asyncio.timeout(45):
            while True:
                begin = json.loads(await ws.recv())
                if begin["type"] == "error":
                    raise RuntimeError(begin)
                if begin["type"] == "session.begin":
                    break
        print("Conversation IDs:", begin["thread_id"], begin["assistant_id"])
        expected = {"encoding": "pcm16", "sample_rate": 24000, "channels": 1}
        if begin["input_format"] != expected or begin["output_format"] != expected:
            raise ValueError("This example expects 24 kHz mono PCM16 in both directions")

        async def send_recording():
            for offset in range(0, len(pcm), 960):
                chunk = pcm[offset:offset + 960]
                await ws.send(chunk)
                await asyncio.sleep(len(chunk) / 48000)  # Pace audio in real time.
            await ws.send(json.dumps({"type": "commit"}))

        async def receive_reply():
            stopping = False
            with open("reply.pcm", "wb") as output:
                while True:
                    event = json.loads(await asyncio.wait_for(ws.recv(), 90))
                    if event["type"] == "audio.delta":
                        output.write(base64.b64decode(event["data"]))
                    elif event["type"] == "transcript.final":
                        print(event["role"], event["text"])
                    elif event["type"] == "tool.call" and not event.get("hosted"):
                        raise RuntimeError("Add a custom tool handler before using this assistant")
                    elif event["type"] == "response.done" and not stopping:
                        stopping = True
                        await ws.send(json.dumps({"type": "stop"}))
                    elif event["type"] == "session.ended":
                        return
                    elif event["type"] == "error":
                        raise RuntimeError(event)

        # If either side fails, cancel the other instead of hanging.
        async with asyncio.TaskGroup() as tasks:
            tasks.create_task(send_recording())
            tasks.create_task(receive_reply())

asyncio.run(main())

Play the saved reply with FFmpeg’s player:

ffplay -f s16le -ar 24000 -ac 1 -nodisp -autoexit reply.pcm

This example saves chunks as they arrive. A live app instead passes those bytes to a playback queue. See the SDK microphone and playback example.

​

5. Use tools during the conversation

Tools work with spoken or typed input. For example, the user can say “Where is order A123?” and the model can call your order lookup before speaking.

​

Internal tools: memory and web search

Add these options to the setup message:

{
  "thread_id": "YOUR_THREAD_ID",
  "provider": "openai",
  "model": "gpt-realtime-mini",
  "memory": "Auto",
  "web_search": "Auto"
}

Then speak, or send a text turn:

{"type":"text","text":"Remember that I prefer short answers. Search the web for today's space news."}

memory: "Readonly" allows searches but not new memories. memory: "Auto" allows both. Use memory_pro instead of memory for PRO memory. Memory is shared by threads on the same assistant; it is not the same as a thread’s history.

If the model chooses web search, you can receive events like:

{"type":"tool.call","tool_call_id":"search-1","name":"search_web","arguments":{"query":"today's space news"},"hosted":true}

{"type":"tool.completed","tool_call_id":"search-1"}

The model receives the tool result and can speak its answer. Do not execute hosted calls or submit outputs for them. Enabling a tool makes it available; it does not guarantee the model will use it. tool.completed ends tool processing; it is not proof that an external search or generation succeeded.

​

Internal tools: images and video

Choose models from the image and video catalogs. Add only the tools you need:

{
  "thread_id": "YOUR_THREAD_ID",
  "provider": "openai",
  "model": "gpt-realtime-mini",
  "image_generation": "auto",
  "image_model_provider": "openrouter",
  "image_model_name": "YOUR_IMAGE_MODEL",
  "video_generation": "auto",
  "video_model_provider": "openrouter",
  "video_model_name": "YOUR_VIDEO_MODEL"
}

Replace the model placeholders with catalog names. image_config and video_config accept settings supported by the selected media model. These tools generate an attachment; they do not send image or video bytes as audio.delta.

{"type":"text","text":"Create an image of a small blue robot, then describe it briefly."}

On successful generation, a media.generated event contains tool_call_id and generated_media. Display those attachments using the same media handling as image tools and video tools. Generation has its own cost and may take longer than a spoken reply. Do not automatically retry a paid generation after a disconnect.

​

External tools: call your own code

First, pass the function in your connection settings. This starts a new conversation and makes a small read-only tool available:

{
  "provider": "openai",
  "model": "gpt-realtime-mini",
  "system_prompt": "Use get_order_status to answer order questions. Never invent a status.",
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_order_status",
      "description": "Look up the delivery status of an order.",
      "parameters": {
        "type": "object",
        "properties": {"order_id": {"type": "string"}},
        "required": ["order_id"],
        "additionalProperties": false
      }
    }
  }]
}

Send this as the first JSON frame on the realtime socket. Add thread_id to continue an existing conversation or assistant_id to start a new one under a known assistant. No separate creation or tool-registration request is needed.

These tools last for this session only. Re-pass them on later connections. If you have saved tools on an assistant, omitting tools inherits them; tools: [] clears them for this session. Explicit memory/search/media selections are added separately.

Next, send a user turn:

{"type":"text","text":"Where is order A123?"}

Receive the call. Normally arguments is an object. Still validate its type and fields before running your function:

{"type":"tool.call","tool_call_id":"order-1","name":"get_order_status","arguments":{"order_id":"A123"},"hosted":false}

Run the function and return the result on the same WebSocket:

# Add this branch inside your receive loop. This is demo data, not a real order API.
if event["type"] == "tool.call" and not event.get("hosted"):
    args = event.get("arguments", {})
    if event["name"] != "get_order_status":
        result = {"error": "Unknown tool"}
    elif not isinstance(args, dict) or not isinstance(args.get("order_id"), str):
        result = {"error": "order_id must be a string"}
    else:
        status = {"A123": "shipped"}.get(args["order_id"])
        result = {"order_id": args["order_id"], "status": status or "not found"}
    await ws.send(json.dumps({
        "type": "tool.outputs",
        "outputs": [{
            "tool_call_id": event["tool_call_id"],
            "name": event["name"],
            "output": result,
        }],
    }))

The wire message looks like this:

{"type":"tool.outputs","outputs":[{"tool_call_id":"order-1","name":"get_order_status","output":{"order_id":"A123","status":"shipped"}}]}

The model can now say “Your order has shipped.” Keep reading for audio.delta, transcripts, or more tool calls. For realtime, use WebSocket tool outputs, not the REST /threads/tool-outputs endpoint used for text runs.

For real database or HTTP work, dispatch the call in a background task so audio reception keeps running. The SDK guide shows this pattern. Check the signed-in user’s access before returning private data. Validate arguments and allow only registered functions; never evaluate code or run a shell command supplied by the model.

​

Multiple calls and cancelled calls

• Calls can arrive in parallel or in several rounds. Match every result to its own tool_call_id and name; do not return the same result twice.
• You may return several results in one outputs array. Every call in it must still be pending and must be a client-run call.
• On tool.cancelled, cancel work for the IDs in tool_call_ids and do not submit their results. A late result for an unknown or cancelled call is rejected.
• If a call fails, return a clear error result instead of inventing a success.
• Do not stop the session on the first response.done in a tool workflow. Let tools finish and the model speak; stop when the user ends the conversation.

{"type":"tool.cancelled","tool_call_ids":["order-1"]}

​

6. End, continue, or change models

When the user presses End conversation, stop capturing audio and send:

{"type":"stop"}

Keep receiving until session.ended or an error, then close the connection. Closing the socket immediately can lose late transcripts. stop is a session shutdown, not a substitute for commit on a manual audio turn. Shutdown allows only a short, bounded wait for late transcripts and usage; it does not wait for every tool. Handle history_incomplete or billing_pending errors rather than assuming everything was saved. Paid media jobs may still finish and save their results after the connection closes.

To continue later, connect with the same thread ID. To change provider or model, finish this session and reconnect with the same thread and a different setup. Only one native realtime session can write to a thread at a time. Neither SDK automatically reconnects or resends unsaved audio. For a fresh conversation, omit thread_id. Keep assistant_id if you want to share that assistant’s memory and defaults, or omit both for a new assistant too.

​

Transcription

The model hears the audio directly. Transcripts let people read the conversation and provide text history for the next session.

• Final transcripts and tool results are saved to the thread. Raw native audio is not archived and there is no automatic downloadable speech URL.
• OpenAI enables input transcription with gpt-4o-mini-transcribe by default. It is separately billed. Disable it with {"audio":{"input":{"transcription":null}}} in provider_options; the model still hears you, but spoken user turns will be missing from saved text history.
• Google enables input and output transcription. xAI provides transcript events from its voice session.
• If transcript.failed arrives, tell the user that part of the conversation could not be saved as text. Do not silently treat it as a complete transcript.

​

Browser and mobile apps

Never put a long-lived API key in browser or mobile code. Your trusted backend should authenticate the user, check their access to the thread, and request a single-use ticket:

curl --fail-with-body https://app.backboard.io/api/threads/realtime/tickets \
  -H "X-API-Key: $BACKBOARD_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{}'

An empty body creates a conversation automatically. Supply thread_id to continue or assistant_id to start fresh under that assistant. The response includes the ticket and both resolved IDs. Return them to that user, not the API key. In the browser:

// ticket and threadId come from your authenticated backend.
const ws = new WebSocket(
  `wss://app.backboard.io/api/threads/realtime?ticket=${encodeURIComponent(ticket)}`,
);
ws.onopen = () => ws.send(JSON.stringify({
  thread_id: threadId,
  provider: "openai",
  model: "gpt-realtime-mini",
}));

Tickets expire after 60 seconds and work once. This snippet only opens the session; your app must still wait for session.begin, capture and resample microphone audio, send PCM, play replies, and clean up on stop. MediaRecorder usually produces WebM/Opus, not the raw PCM this default session expects.

​

Common problems

​

Usage and billing

Native audio uses provider-cost billing with no markup. A stored BYOK key takes precedence for that provider. Hosted memory, search, and media tools have separate charges. Free trial credits do not fund native audio; use eligible purchased or subscription credits, or BYOK.

Audio tokens or duration, text, cached context, and optional transcription can all contribute to cost. The transcript’s word count is not the full usage. See the model catalog and billing activity. Disconnecting does not cancel usage already incurred.

For provider voices, configuration updates, advanced formats, and time limits, see the API reference.