Speech session (WebSocket)

Overview

Our ultra-low latency, full-duplex speech session API is ideal for applications like voice assistants and chatbots that need to be snappy and/or don’t have all the text upfront.

Protocol

Our streaming endpoint uses a bidirectional protocol that sends both text and binary data. The protocol is based on the WebSocket Protocol and WebSocket API.

First message

The first message sent to the server must be a JSON object with the following fields:

X-API-Key

string

required

Your API key; get it from your account page.

voice

string

required

The voice id of the voice to use for synthesis. Voice ids can be retrieved by a call to List voices.

format

string

default:"mp3"

The desired output audio format. One of:

mp3: 96kbps MP3 audio. This format is useful for applications that need to play the audio directly to the user.
pcm_s16le: PCM signed 16-bit little-endian audio.
pcm_f32le: PCM 32-bit floating-point little-endian audio.
ulaw: 8-bit G711 µ-law audio with a WAV header. This format is most useful for telephony applications.
webm: WebM format with Opus audio codec. This format is ideal for web browsers and modern streaming applications that prioritize efficient bandwidth usage.

language

str

default:"auto"

The desired language of the synthesized speech. Two letter ISO 639-1 code. Defaults to language auto-detection. Find the list of available languages here.

sample_rate

integer

default:"24000"

The desired output audio sample rate. One of:

24000: 24kHz audio. This sample rate is useful for applications that need high-quality audio.
16000: 16kHz audio. This sample rate is useful for applications that need to save bandwidth.
8000: 8kHz audio. This sample rate is most useful for telephony applications and µ-law encoding.

return_extras

boolean

default:"false"

Controls whether the server will return extra information about the synthesis. This information includes durations, buffer_empty, and warnings. See the Receiving Extras section for more information.

{
    "X-API-Key": "<LMNT_API_KEY>",
    "voice": "morgan",
    "format": "mp3",
    "sample_rate": 24000,
    "return_extras": true
}

Sending text

After the first message, you can send text to the server as a JSON object:

{"text": "Hello, world!"}

The text you send can be split at any point. For example, sending:

{"text": "This is a test of the emergency broadcast system"}

is semantically equivalent to sending these two messages:

{"text": "This is a test of the eme"}
{"text": "rgency broadcast system"}

Flushing (trigger synthesis)

If you want to force the server to synthesize the text that it has without closing the connection, you can send a JSON object with the flush field set to true:

{"flush": true}

You will be notified when the server has finished synthesizing all the text that it has by the buffer_empty field in the extra information. See the Receiving Extras section for more information.

Be careful when using flush. Our models are designed to factor in context when synthesizing audio. When flushing the buffer at arbitrary points, your speech may sound less natural.

The last message

If you are done sending text to the server, you can close the connection by sending a JSON object with the eof field set to true. This will cause the server to synthesize all the text it has and then close the connection.

{"eof": true}

Receiving audio

Once the server has received enough text, the server will respond with chunks of 96kbps mono MP3 audio with a sampling rate of 24kHz. As more text is streamed to the server, the server will continue to send more audio chunks. The audio chunks are sent as binary data.

To produce the most natural-sounding speech, our API waits for roughly two full sentences to be sent before synthesizing audio. If you want to receive audio before this threshold is met, you can use the flush field to force the server to synthesize the text it has.

Receiving extras

If you set the return_extras field to true in the first message, the server will also send extra information about each synthesized chunk. This information is sent as a serialized JSON object (string) and will be sent before its corresponding audio chunk. The extra information includes:

durations

array of duration objects

An array of objects that detail the duration of each text token in the synthesized chunk. This information is useful for applications that need to synchronize the synthesized audio with the text that was sent to the server. The format of each object is described below.

Show properties

Each object describes the duration of a chunk of text (e.g., words, punctuation, and spaces) with the following keys:

text

string

The text itself.

start

number

The time at which the text starts, in seconds

duration

number

The overall duration of the text, in seconds

The durations array resets its start time for each chunk of audio.

buffer_empty

boolean

Indicates whether the server has finished synthesizing all the text that it has received. This is useful for applications that want to know when the server has finished synthesizing all the text that it has without closing the connection.

warning

string

Contains any warnings that the server has encountered during synthesis, such as exceeding the number of free characters.

Here is an example of the extra information that the server sends:

{
  "durations": [
    {
      "text": "",
      "start": 0,
      "duration": 0.2
    },
    {
      "text": "Using",
      "start": 0.2,
      "duration": 0.4
    },
    {
      "text": " ",
      "start": 0.6,
      "duration": 0.025
    },
    {
      "text": "LMNT",
      "start": 0.625,
      "duration": 0.425
    },
    {
      "text": "",
      "start": 1.05,
      "duration": 0.025
    }
    ...
  ]
  "buffer_empty": false,
  "warning": "string"
}

Note that the extra data JSON is always sent before the audio chunk that it corresponds to. Take care to interpret incoming data correctly. Audio is sent as bytes and extra data is sent as a string.

Errors

If the server encounters an error, it will close the connection with a reason. To make debugging easier, the server also send a JSON object with a descriptive message in a string field: error before closing the connection.

{"error": "string"}

Examples

Take a look at our Python SDK source code and Node SDK source code.

Speech

Voice

Account

​Overview

​Protocol

​First message

​Sending text

​Flushing (trigger synthesis)

​The last message

​Receiving audio

​Receiving extras

​Errors

​Examples

Overview

Protocol

First message

Sending text

Flushing (trigger synthesis)

The last message

Receiving audio

Receiving extras

Errors

Examples