The socket opens at /realtime?session=rts_… with the session that POST /agents/v1/realtime/sessions returns; the secret key never travels over it. Every message is a JSON object with type.
The client sends
Turns (text or audio), instruction changes and the end of the conversation.
session.update
Change the agent’s instructions mid-conversation.
{
"type": "object",
"required": [
"type"
],
"additionalProperties": false,
"properties": {
"type": {
"const": "session.update"
},
"instructions": {
"type": "string",
"maxLength": 500
}
}
}input_text.send
A complete user turn, as text.
{
"type": "object",
"required": [
"type",
"text"
],
"additionalProperties": false,
"properties": {
"type": {
"const": "input_text.send"
},
"text": {
"type": "string",
"minLength": 1,
"maxLength": 4000
}
}
}input_audio.append
A chunk of the user’s audio.
{
"type": "object",
"required": [
"type",
"audio"
],
"additionalProperties": false,
"properties": {
"type": {
"const": "input_audio.append"
},
"audio": {
"type": "string",
"contentEncoding": "base64",
"description": "PCM 16-bit little-endian, 16 kHz, mono."
}
}
}input_audio.commit
The user stopped talking; the agent answers what it heard.
{
"type": "object",
"required": [
"type"
],
"additionalProperties": false,
"properties": {
"type": {
"const": "input_audio.commit"
}
}
}session.end
Hang up.
{
"type": "object",
"required": [
"type"
],
"additionalProperties": false,
"properties": {
"type": {
"const": "session.end"
}
}
}The server sends
The session start, the agent’s answer in deltas, running usage, errors and the end.
session.started
{
"type": "object",
"required": [
"type",
"session"
],
"properties": {
"type": {
"const": "session.started"
},
"session": {
"type": "object",
"required": [
"id",
"workspace",
"cell",
"livemode",
"input_audio_format",
"max_duration_seconds"
],
"properties": {
"id": {
"type": "string"
},
"workspace": {
"type": "string"
},
"cell": {
"type": "string"
},
"livemode": {
"type": "boolean"
},
"input_audio_format": {
"type": "string",
"const": "pcm16_16khz_mono"
},
"max_duration_seconds": {
"type": "integer"
},
"served_by": {
"type": "string",
"description": "The service and language that answered (agents-runtime/python)."
}
}
}
}
}response.text.delta
The next piece of the agent’s answer.
{
"type": "object",
"required": [
"type",
"response_id",
"delta"
],
"properties": {
"type": {
"const": "response.text.delta"
},
"response_id": {
"type": "string"
},
"delta": {
"type": "string"
}
}
}response.done
{
"type": "object",
"required": [
"type",
"response_id",
"text"
],
"properties": {
"type": {
"const": "response.done"
},
"response_id": {
"type": "string"
},
"text": {
"type": "string"
}
}
}usage.updated
Running totals, after every turn.
{
"type": "object",
"required": [
"type",
"elapsed_seconds",
"audio_seconds",
"turns"
],
"properties": {
"type": {
"const": "usage.updated"
},
"elapsed_seconds": {
"type": "number"
},
"audio_seconds": {
"type": "number"
},
"turns": {
"type": "integer"
}
}
}session.ended
{
"type": "object",
"required": [
"type",
"reason",
"billed_minutes"
],
"properties": {
"type": {
"const": "session.ended"
},
"reason": {
"type": "string",
"enum": [
"client_ended",
"max_duration",
"client_disconnected"
]
},
"billed_minutes": {
"type": "integer",
"description": "0 in test mode."
},
"usage_record_id": {
"type": [
"string",
"null"
],
"description": "The billing record (contracts/internal/proto/qomni/billing/v1), when billed."
}
}
}error
{
"type": "object",
"required": [
"type",
"error"
],
"properties": {
"type": {
"const": "error"
},
"error": {
"type": "object",
"required": [
"code",
"message"
],
"properties": {
"code": {
"type": "string",
"description": "invalid_message, unknown_message_type, session_expired…"
},
"message": {
"type": "string"
}
}
}
}
}