Getting and sending WebSocket messages

Your WebSocket server must validate incoming messages from Twilio using the X-Twilio-Signature header. For detailed guidance on setting up signature validation, see Configure your WebSocket server. The messages that you receive from Twilio will be one of the following types:

Conversation Relay sends this message immediately after establishing the WebSocket connection. The customParameters field contains any parameters you defined using <Parameter> child elements inside your <ConversationRelay> TwiML.

1
{
2
  "type": "setup",
3
  "sessionId": "VX00000000000000000000000000000000",
4
  "accountSid": "ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
5
  "parentCallSid": "",
6
  "callSid": "CA00000000000000000000000000000000",
7
  "from": "+18005550100",
8
  "to": "+18005550101",
9
  "forwardedFrom": "+18005550102",
10
  "callType": "PSTN",
11
  "callerName": "",
12
  "direction": "inbound",
13
  "callStatus": "RINGING",
14
  "customParameters" : {
15
    "callReference": "bar"
16
  }
17
}

When you set applicationReconnect="true" in your TwiML, the setup message also includes the following fields:

To learn how to handle a reconnect, see Handling connection drops.

Conversation Relay sends this message when the caller says something. The last field indicates whether the caller speech has completed based on the report from the speech-to-text provider and the configuration of this ConversationRelay session. For example, when speechTimeout is enabled, last will be sent after the specified speechTimeout is reached.

How the lang field is populated depends on your session's speech-to-text (STT) configuration:

• Automatic Language Detection (transcriptionLanguage="multi"): The underlying STT provider uses a multilingual model to dynamically classify caller speech per utterance. The detected BCP-47 tag is then attached to lang in real time.
• Static Language Setting: If a fixed language is configured for transcription (e.g., transcriptionLanguage="en-US"), lang echoes that configured language tag for every prompt message.

You can use the lang value in your application to dynamically inform your LLM system prompt or trigger Text-to-Speech (TTS) voice changes mid-conversation.

1
{
2
  "type": "prompt",
3
  "voicePrompt": "Hi! Can you tell me about life?",
4
  "lang": "en-US",
5
  "last": true
6
}

Conversation Relay sends this message when you turn on DTMF detection and the caller presses a key.

1
{
2
  "type": "dtmf",
3
  "digit": "1"
4
}

Conversation Relay sends this message when the caller interrupts TTS playback by speaking.

1
{
2
  "type": "interrupt",
3
  "utteranceUntilInterrupt": "Life is a complex set of",
4
  "durationUntilInterruptMs": 460
5
}

Conversation Relay sends this message when you subscribe to the config-updated event in the events attribute of your <ConversationRelay> TwiML. Conversation Relay sends it at session startup and whenever a configuration change takes effect.

1
{
2
  "type": "info",
3
  "name": "configUpdated",
4
  "value": {
5
    "session": {
6
      "ttsLanguage": "es-ES"
7
    }
8
  }
9
}

The value object has the same shape as a config message. To learn how Conversation Relay reports configuration changes, see Update configuration during a call.

Conversation Relay sends this message when an error occurs during the session.

1
{
2
  "type": "error",
3
  "description": "Invalid message received, code: 64107, type: config, error: Unexpected fields in 'session': [foo]"
4
}

Conversation Relay validates the messages that your application sends. A message that it can't parse, or one with unknown fields or invalid values, returns error 64107. An invalid interruptible or preemptible value returns error 64101. Rejected config, hints, and language updates return an error message with a description and no code. None of these errors end the session, and Conversation Relay doesn't process the message.

Send text tokens messages to Twilio and Conversation Relay will convert them to speech. We recommend streaming text tokens as soon as they are available, rather than waiting for the entire message to be ready. This reduces latency. Use the last attribute to indicate the final token in the talk cycle.

1
{
2
  "type": "text",
3
  "token": "Hello world!",
4
  "last": false,
5
  "interruptible": false,
6
  "preemptible": false
7
}

To fine-tune synthesized speech, you can passthrough SSML tags within the token to provide the pronounciation of a word or an acronym, specify where pauses should be, or increase or decrease the speed of spoken text. Supported SSML tags depend on the active ttsProvider for the session.

If the ttsProvider is Google or Amazon, see the Speech Synthesis Markup Language (SSML) documentation for the list of supported tags.

If the ttsProvider is ElevenLabs and you use v3 or v4, SSML isn't supported. Use audio tags instead. See Use audio tags with ElevenLabs v3 and v4. For earlier ElevenLabs models, the language must be en-US, and only the <phoneme> tag is supported. See the ElevenLabs documentation for guidance on defining phoneme tags. Example:

1
{
2
  "type": "text",
3
  "token": "Hello from <phoneme alphabet=\"ipa\" ph=\"ˈtwɪlioʊ\">Twilio</phoneme>.",
4
  "last": false,
5
  "interruptible": false,
6
  "preemptible": false
7
}

Request to play media to the caller.

1
{
2
  "type": "play",
3
  "source": "https://api.twilio.com/cowbell.mp3",
4
  "loop": 1,
5
  "preemptible": false,
6
  "interruptible": true
7
}

Request to send DTMF digits to the caller. Conversation Relay sends digits as per Twilio's <Play> digits attribute.

1
{
2
  "type": "sendDigits",
3
  "digits": "9www4085551212"
4
}

Change the transcription and TTS language during the session. This affects future TTS and STT sessions.

1
{
2
  "type": "language",
3
  "ttsLanguage": "sv-SE",
4
  "transcriptionLanguage": "en-US",
5
  "hints": "faktura,saldo,betalning"
6
}

This message is shorthand for a config message that sets ttsLanguage and transcriptionLanguage, and the settings of those languages. To switch to a language that the session doesn't have, give each part that you switch in full: ttsProvider and voice, or transcriptionProvider and speechModel. See Add a language and switch to it.

Update language and session settings during the conversation without interrupting the call. An update changes only the settings that you include. Every setting that you leave out keeps its current value.

1
{
2
  "type": "config",
3
  "language": {
4
    "code": "en-US",
5
    "speechModel": "nova-3-general",
6
    "hints": "invoice,autopay"
7
  },
8
  "session": {
9
    "dtmfDetection": true,
10
    "interruptible": "speech"
11
  }
12
}

If any value in the message is invalid, Conversation Relay rejects the entire message and the call continues with the previous configuration. For the list of settings that you can update, when each update takes effect, and how to troubleshoot errors, see Update configuration during a call.

Update the speech recognition hints for the active transcription language. This message is shorthand for a config message that sets hints on the active language.

1
{
2
  "type": "hints",
3
  "hints": "invoice,balance,autopay,routing number"
4
}

End the session and return control of the call to Twilio through Conversation Relay.

1
{
2
  "type": "end",
3
  "handoffData": "{\"reasonCode\":\"live-agent-handoff\", \"reason\": \"The caller wants to talk to a real person\"}"
4
}

For errors, such as messages that Conversation Relay doesn't understand, we will respond with an error message.

If your WebSocket sends unidentified messages to Conversation Relay and the last 10 messages remain unidentified, we will terminate the connection. The status code will be 1007 with the reason "Too many consecutive malformed messages." In that case, we will report an error 64105 "WebSocket Ended."

The following error codes are specific to Conversation Relay. The Ends session column indicates whether the error ends the ConversationRelay session.

Note: *This error code covers both warning-level and error-level events. While minor configuration issues (like DTMF detection arguments) may allow the session to continue, critical failures—such as the inability to play or synthesize audio due to invalid arguments—will terminate the session.

If the WebSocket disconnects unexpectedly and you haven't set applicationReconnect="true" in your TwiML, Conversation Relay doesn't reconnect, and the call disconnects with a failed status.

When you set applicationReconnect="true" on your <ConversationRelay> noun, Conversation Relay keeps the call active if the WebSocket drops. It buffers up to 500 inbound events (or up to 10 seconds of call history), opens a new WebSocket connection to your url, and sends a setup message with connectionType set to reconnect. It then replays the buffered events. To learn how to restore your session state, see Handling connection drops.

If you don't use applicationReconnect, or if automatic reconnection doesn't restore the session, implement reconnection logic by initiating a new <Connect><ConversationRelay> request:

• Re-establish the connection: If you lose the WebSocket connection, handle the disconnect in your <Connect> element's action URL callback by returning new TwiML containing <Connect><ConversationRelay> to restore the session.
• Validate call consistency in Conversation Relay: Ensure the callSid remains the same to confirm continuity of the original call session.

This approach helps maintain session stability and consistency following any connection disruptions.

This guide teaches the basics required for the following use cases:

You can use this guide to build intelligent, conversational voice agents that interact with callers in real-time. By processing WebSocket messages, your AI can understand speech and respond dynamically. To learn more advanced features that you can use with AI agents, see AI agents.

You can use this guide to capture and process real-time audio data for advanced machine learning workflows. This allows you to generate live transcriptions or perform sentiment analysis during a call.

To learn more advanced features that you can use with AI/ML transcription, see Voice AI/ML transcription.

After following this guide, you can successfully exchange real-time JSON messages between your server and Twilio via a WebSocket. You can confirm it works by verifying that your server receives a setup message upon connection and can successfully trigger text-to-speech by sending a text token message.

Explore the following guides to build on what you've learned in this guide:

• Picking a voice: Configure different text-to-speech voices and providers for your session.
• Update configuration during a call: Change language, voice, and session settings with config messages.
• Consume a real-time Media Stream using WebSockets, Python, and Flask: Handle raw audio streams using a similar WebSocket architecture.

Find reference documentation for the topics on this page.

• <ConversationRelay> TwiML noun
• TwiML™ Voice: <Connect>
• TwiML™ Voice: <ConversationRelay>
• TwiML™ Voice: <Parameter>
• Text-to-Speech (TTS)