Realtime Audio

​

Connect

wss://app.backboard.io/api/threads/realtime

Authenticate with the X-API-Key header. Send a JSON text frame containing the session configuration, then wait for session.begin before sending audio. Provider setup/history events can arrive before session.begin; keep reading until it arrives. API keys in query parameters are not accepted on this endpoint. For working clients, see the Python and Node.js guide.

{
  "provider": "openai",
  "model": "gpt-realtime-mini",
  "provider_options": {
    "audio": {"output": {"voice": "marin"}}
  },
  "memory": "off"
}

No separate assistant/thread creation request is needed. Omit both IDs to create a fresh conversation, send thread_id to continue one, or send only assistant_id to start a new thread under that assistant. session.begin returns both IDs. When both IDs are supplied, the thread takes precedence. This is the same ID lifecycle as POST /threads/messages, with a WebSocket for two-way audio. Choose a supported model from GET /models?model_type=realtime.

​

Session parameters

Values are case-sensitive: memory and search use Auto; media generation uses auto. Tool selection applies to the session. Reconnect to apply a different selection. Session overrides are not saved to the assistant; re-pass them on later connections. Native instruction fields in provider_options take precedence over system_prompt.

​

Provider settings

Put native fields directly inside provider_options—do not add another session, setup, or provider-name wrapper. Supplied fields override defaults; unspecified nested fields retain their defaults. Settings must be supported by your selected model. provider_options cannot change the selected model.

• OpenAI

• Google

• xAI

{
  "thread_id": "YOUR_THREAD_ID",
  "provider": "openai",
  "model": "gpt-realtime-mini",
  "provider_options": {
    "instructions": "Speak warmly and keep replies under two sentences.",
    "max_output_tokens": 1024,
    "audio": {
      "input": {
        "format": {"type": "audio/pcm", "rate": 24000},
        "turn_detection": {"type": "semantic_vad"},
        "transcription": {"model": "gpt-4o-mini-transcribe", "language": "en"}
      },
      "output": {
        "voice": "cedar",
        "speed": 1.1,
        "format": {"type": "audio/pcm", "rate": 24000}
      }
    }
  }
}

Custom instructions override the assistant’s session instructions. Leave them unset to use the assistant configuration. Provider-native functions in tools are added alongside assistant and Backboard tools; they cannot replace an existing function with the same name. Provider-hosted tools and provider-stored prompts or resumed sessions may require a stored BYOK key. Prefer the Backboard tool parameters above for credit-funded sessions.

​

Transcription settings

OpenAI’s input transcription is optional and separately billed. To keep voice conversation but disable the saved user transcript:

{"audio": {"input": {"transcription": null}}}

Use that object as provider_options. Alternatively, change the transcription model or language in audio.input.transcription. Input transcription does not route the spoken conversation through a text model. See history and transcripts.

​

Audio and events

session.begin includes session_id, thread_id, input_format, and output_format:

{
  "type": "session.begin",
  "session_id": "SESSION_ID",
  "thread_id": "THREAD_ID",
  "assistant_id": "ASSISTANT_ID",
  "input_format": {"encoding": "pcm16", "sample_rate": 24000, "channels": 1},
  "output_format": {"encoding": "pcm16", "sample_rate": 24000, "channels": 1}
}

Send audio as binary frames matching input_format. For PCM16, use signed 16-bit little-endian samples, not a WAV/MP3 file. Output audio.delta.data is base64-encoded audio; decode it and play using the event’s format. Receive events while sending microphone frames rather than waiting for the microphone to finish.

Format fields on each audio event are flat:

{"type":"audio.delta","data":"BASE64_AUDIO_BYTES","item_id":"reply-1","encoding":"pcm16","sample_rate":24000,"channels":1}

BASE64_AUDIO_BYTES is a placeholder. Use real decoded bytes for playback. Raw provider.event messages can describe the same audio; do not play both copies.

​

Client events

​

Received events

Gemini Extended Thinking can produce multiple utterances in one interaction. Do not stop on its first response.done; use provider.event.event.serverContent.interactionStatus === "IDLE" when you need to detect the end of that interaction.

​

Custom tool results

Run only tool.call events with hosted: false. Validate the function and its arguments, then return tool.outputs on this socket. Each result must match a pending tool_call_id and its exact name. Return one result per call; a batch can contain up to 32. Do not return hosted, cancelled, duplicate, or already completed calls. Client-run tools do not get a tool.completed acknowledgement.

This is not the REST tool-output protocol. See the complete tool round trip and SDK handlers.

​

Graceful stop

stop does not commit buffered input or wait for every tool. It allows a bounded wait for late transcripts and usage before finalization. Keep receiving until session.ended or failure; handle history_incomplete and billing_pending errors. Paid image/video work can continue and save results after disconnect.

​

Change settings during a session

Use provider.event for updates supported by the provider. For example, update OpenAI instructions without reconnecting:

{
  "type": "provider.event",
  "event": {
    "type": "session.update",
    "session": {"instructions": "Keep answers to one sentence."}
  }
}

Not every setting is mutable after connection. Reconnect to change the provider, model, or Backboard tool selection. Provider errors identify invalid settings.

​

Limits and reconnecting

• Each client frame may be at most 256 KiB. Stream small audio chunks.
• Send the initial configuration within 30 seconds of opening the connection.
• Sessions last up to 30 minutes and accept up to 100 MiB of client input.
• A connection with no client audio or events for 120 seconds times out. Silence while microphone frames continue arriving is not inactivity.
• One native realtime session may write to a thread at a time. Finish it before reconnecting.
• History is limited to 200 messages after any saved summary and 200,000 characters of serialized history plus assistant instructions. Exceeding these limits rejects setup; it does not silently truncate the conversation.

On a disconnect, reconnect with the same thread_id to continue from saved history. There is no automatic reconnect or replay of unsaved microphone audio. Increasing an SDK connection timeout does not change these session limits.

​

Browser connections

Do not expose a long-lived API key in browser code. Obtain a single-use ticket through a trusted application endpoint that authenticates and authorizes the user:

curl -X POST https://app.backboard.io/api/threads/realtime/tickets \
  -H "X-API-Key: $BACKBOARD_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{}'

Use {} to auto-create a conversation, {"assistant_id":"YOUR_ASSISTANT_ID"} for a new thread under an existing assistant, or {"thread_id":"YOUR_THREAD_ID"} to continue a conversation. The response includes ticket, expires_in (60 seconds), websocket_path, thread_id, and assistant_id. Connect the browser to wss://app.backboard.io/api/threads/realtime?ticket=URL_ENCODED_TICKET, then send the same setup frame. Each ticket can be used once. Browser clients supply their own microphone capture and audio playback. The socket may omit thread_id when using a ticket: it uses the ticket’s resolved thread. A ticket cannot create or switch to a different conversation. The Chat dashboard may instead supply chat_thread_id when requesting a ticket; do not combine it with the public thread or assistant identifiers. The ticket is bound to the thread and, when present, the Origin header on the ticket request. A backend issuing tickets should deliberately choose the intended browser origin. websocket_path excludes the deployment’s /api prefix.