Generative Custom Operators

Generative Custom Operators apply LLM-backed language analysis to Voice Intelligence Transcripts. This guide assumes that you have already:

1. Created a Voice Intelligence Service.
2. Set up Transcription of Voice Calls either through auto-transcribe or by using the REST API.
3. [Optional] Have one or more completed transcripts in your Voice Intelligence Service.

Completing Step 3 is strongly recommended as it will allow you to use the preview results feature of Generative Custom Operators – testing your draft prompt with previous transcripts before saving your new Language Operator.

Expand image

The simplest way to create a Generative Custom Operator is using the Twilio Console:

• Navigate to the Voice Intelligence product
• Click Services in the sidebar
• Select an existing Voice Intelligence Service, or create a new one
  • Note: The Service you select should be the same service containing the transcripts you'd like the Generative Custom Operator to analyze
• Click the Create Custom Operator button
• Select Generative as your operator type

This will launch the Generative Custom Operator creation wizard in Console, guiding you through the creation process.

The prompt for your Generative Custom Operator provides instructions for the LLM to analyze incoming transcripts. In the textbox, explain what you would like to achieve, being specific and including key requirements for how you'd like the LLM to behave.

Please see the sections on:

• Best Practices for prompt writing
• Example Use Cases for more inspiration on the types of prompts you can write.

Important notes on prompting:

• The maximum allowable prompt length is 4,000 characters, or roughly 600-1000 words
• Generative Custom Operators can be used to analyze transcripts in all languages supported by Voice Intelligence
• In addition, your prompt can be written in a different language than the transcripts it will analyze. For example, while the transcripts might be in Spanish, you can write the prompt in English

The Console wizard for Generative Custom Operators provides the ability to preview operator results before it is finalized. This is a useful way to test the prompt you've written, ensuring that it yields the intended results.

Expand image

To preview your Generative Custom Operator, find the preview operator results pane on the right side of the Console wizard. Click the choose test transcript button and choose a previously transcribed conversation to use to test your prompt. Be sure to choose a transcript that is as close as possible to the type of interaction you would expect to happen in production, once the Operator has been created. This will give the most realistic preview of "real-world" operator results.

Once a desired test transcript is selected, click the preview operator results button to run your Operator. Once complete, you will see the results returned by the LLM.

If the results weren't what you expected:

1. Iterate on your prompt: Revise your prompt, being sure to take into consideration our prompting best practices.
2. Try a different transcript: It may be the underlying conversation data itself that is yielding a particular operator result. A different transcript may produce different results.
3. Use Advanced Settings: Advanced Settings provide additional options for customizing operator results, including output formatting controls and providing training examples to the LLM.

Once you're content with your prompt, click Next and review the summary of your Generative Custom Operator. If everything looks good, click Submit to save your new Operator.

The final step is to add your new Generative Custom Operator to a Voice Intelligence Service. This will allow you to use the Operator to analyze transcripts in that Service.

Upon successful creation of your Generative Custom Operator, you will be redirected to the Operator list page for your service.

To add your new Operator to a Service:

• Find your recently created Operator in the list. It will be labeled with type equal to Generative operator
• Click +Add to apply it to your Service

Great! Your new Generative Custom Operator will now execute with each new Transcript uploaded to this Voice Intelligence Service.

The Intelligence REST API can be used to create Generative Custom Operators in addition to the Console. There are a few key parameters to include when creating a Generative Custom Operator:

POST https://intelligence.twilio.com/v2/Operators/Custom

Encoding type: application/x-www-form-urlencoded

Request Parameters:

Below is a simple example of a request to create a basic Generative Custom Operator using the API:

1
// Download the helper library from https://www.twilio.com/docs/node/install
2
const twilio = require("twilio"); // Or, for ESM: import twilio from "twilio";
3

4
// Find your Account SID and Auth Token at twilio.com/console
5
// and set the environment variables. See http://twil.io/secure
6
const accountSid = process.env.TWILIO_ACCOUNT_SID;
7
const authToken = process.env.TWILIO_AUTH_TOKEN;
8
const client = twilio(accountSid, authToken);
9

10
async function createCustomOperator() {
11
  const customOperator = await client.intelligence.v2.customOperators.create({
12
    config: {
13
      prompt:
14
        "You are labeling the transcripts of calls between a customer and a virtual agent.  Your job is to label the transcript below with one of these labels based on the description\nLabels\nWARM -  A customer who has shown interest in our products and offerings. They may not be open to schedule an appointment at this point or commit to more specifics.\nHOT - A qualified lead is someone who understands what we can offer and are willing to commit. They are more likely to result in a sale. A customer who schedules an appointment is also considered HOT lead.\nCOLD - A cold lead is the one that is not interested or has lost interest in the product and offering.\n",
15
    },
16
    friendlyName: "LeadGeneration",
17
    operatorType: "Generative",
18
  });
19

20
  console.log(customOperator.accountSid);
21
}
22

23
createCustomOperator();

1
{
2
  "account_sid": "ACaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
3
  "sid": "LYaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
4
  "friendly_name": "LeadGeneration",
5
  "description": "New Operator",
6
  "author": "ACaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
7
  "operator_type": "Generative",
8
  "version": 1,
9
  "availability": "public",
10
  "config": {
11
    "configuration": {
12
      "field": "value"
13
    }
14
  },
15
  "date_created": "2010-08-31T20:36:28Z",
16
  "date_updated": "2010-08-31T20:36:28Z",
17
  "url": "https://intelligence.twilio.com/v2/Operators/Custom/LYaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
18
}

Once the Generative Custom Operator has been created via the API, it must also be added to a Voice Intelligence Service to be used for analyzing transcripts.

Please see the Operator Attachment API docs for instructions on how to do this.

You will need the OperatorSid of the newly created Operator to attach it to a Service. This can be found in the response to the Operator creation API request above.

Advanced settings provide increased configurability for Generative Custom Operators. Two advanced settings are available: JSON Output Format and Training Examples.

Generative Custom Operators allow you to specify the output format that best suits your needs. By default, the LLM will return an unstructured text response. However, you can optionally choose to structure the Operator Result output by providing a JSON schema that the LLM will use to format the results. JSON schema is a powerful standard for defining the structure of the data you expect to receive from the LLM. It allows you to specify the property names and data types for each property, as well as any constraints or requirements for the properties.

When using JSON output format, the LLM will return results that adhere to the schema you provide. This can be useful for ensuring that the output is structured in a way that is easy to consume by downstream systems or for more complex analysis.

At minimum, a valid JSON results schema will include the following keywords:

• type : Must be set to "object"
• properties : An object containing the property names and their data types you would like the LLM to return

Here's a simple example of a valid JSON object used for a tailored summarization use case that can be passed as json_result_schema parameter when creating the GenerativeJSON operator:

1
{
2
    "type": "object",
3
    "properties": {
4
        "budget": { "type": "string" },
5
        "authority": { "type": "string" },
6
        "need": { "type": "string" },
7
        "timeline": { "type": "string" }
8
    }
9
}

Additional notes on JSON output formatting:

• The root level type of a JSON schema must be set to object
• The following property data types are supported : string, number, boolean, integer, object, array, enum, anyOf
• Max 100 object properties and 5 levels of nesting are supported
• Max 500 enum values across all enum properties are supported
• Notable JSON Schema keywords not supported include:
  • For strings: minLength, maxLength, pattern, format
  • For numbers: minimum, maximum, multipleOf
  • For objects: patternProperties, unevaluatedProperties, propertyNames, minProperties, maxProperties
  • For arrays: unevaluatedItems, contains, minContains, maxContains, minItems, maxItems, uniqueItems
• Structured Operator Results will be returned in the same order as the ordering of keys in the schema
• In the event an Operator execution request is refused for safety reasons the Operator Result API response will include a new field called refusal to indicate that the LLM refused to fulfill the request
• Twilio will automatically set additionalProperties to false and specify all provided fields as required (constraints of Structured Outputs). You don't need to pass these fields as part of your JSON schema. Twilio will automatically overwrite any user-provided values for these fields.

To use the REST API to create a Generative Custom Operator with JSON output, in addition to the standard fields, you must:

• Set OperatorType to GenerativeJSON.
• Set Config.json_result_schema parameter to the JSON schema you'd like the LLM to adhere to.

An example request can be found below:

1
// Download the helper library from https://www.twilio.com/docs/node/install
2
const twilio = require("twilio"); // Or, for ESM: import twilio from "twilio";
3

4
// Find your Account SID and Auth Token at twilio.com/console
5
// and set the environment variables. See http://twil.io/secure
6
const accountSid = process.env.TWILIO_ACCOUNT_SID;
7
const authToken = process.env.TWILIO_AUTH_TOKEN;
8
const client = twilio(accountSid, authToken);
9

10
async function createCustomOperator() {
11
  const customOperator = await client.intelligence.v2.customOperators.create({
12
    config: {
13
      prompt:
14
        "You are labeling the transcripts of calls between a customer and a virtual agent.  Your job is to label the transcript below with one of these labels based on the description\nLabels\nWARM -  A customer who has shown interest in our products and offerings. They may not be open to schedule an appointment at this point or commit to more specifics.\nHOT - A qualified lead is who understands what we can offer and are willing to commit. They are more likely to result in a sale. A customer who schedules an appointment is also considered HOT lead.\nCOLD - A cold lead is the one that is not interested or has lost interest in the product and offering.\n",
15
      json_result_schema: {
16
        type: "object",
17
        properties: {
18
          label: {
19
            type: "string",
20
          },
21
        },
22
      },
23
    },
24
    friendlyName: "SalesLeadClassification",
25
    operatorType: "GenerativeJSON",
26
  });
27

28
  console.log(customOperator.accountSid);
29
}
30

31
createCustomOperator();

In addition to specifying the output format, you can also provide training examples. Examples can help improve Operator results by contextualizing and clarifying as well as signaling intent and adding specificity to your request. These examples should consist of a sample input and the corresponding expected output. The LLM will use these examples to better understand the task you are asking it to perform.

Training examples can be passed in the Config object of either Generative or GenerativeJSON Custom Operators. The examples parameter should be an array of objects, each containing an input and output key:

• input: Provide a transcript snippet for the LLM to use as an example conversation
• output: Provide what you would expect the LLM to output for the given transcript snippet

Below is an example of a simple examples array:

1
"examples": [
2
    {
3
        "input": "This is an example sentence which qualifies the lead as HOT.",
4
        "output": "HOT"
5
    }
6
]

An example full request to the API can be found below:

1
// Download the helper library from https://www.twilio.com/docs/node/install
2
const twilio = require("twilio"); // Or, for ESM: import twilio from "twilio";
3

4
// Find your Account SID and Auth Token at twilio.com/console
5
// and set the environment variables. See http://twil.io/secure
6
const accountSid = process.env.TWILIO_ACCOUNT_SID;
7
const authToken = process.env.TWILIO_AUTH_TOKEN;
8
const client = twilio(accountSid, authToken);
9

10
async function createCustomOperator() {
11
  const customOperator = await client.intelligence.v2.customOperators.create({
12
    config: {
13
      prompt:
14
        "You are labeling the transcripts of calls between a customer and a virtual agent.  Your job is to label the transcript below with one of these labels based on the description\nLabels\nWARM -  A customer who has shown interest in our products and offerings. They may not be open to schedule an appointment at this point or commit to more specifics.\nHOT - A qualified lead is someone who understands what we can offer and are willing to commit. They are more likely to result in a sale. A customer who schedules an appointment is also considered HOT lead.\nCOLD - A cold lead is the one that is not interested or has lost interest in the product and offering.\n",
15
      examples: [
16
        {
17
          input: "This is an example sentence which qualifies the lead as HOT.",
18
          output: "HOT",
19
        },
20
      ],
21
    },
22
    friendlyName: "LeadGeneration",
23
    operatorType: "Generative",
24
  });
25

26
  console.log(customOperator.accountSid);
27
}
28

29
createCustomOperator();

In order to ensure the best possible results from your Generative Custom Operators, it is helpful to use prompting best practices. The prompt is the instructions for the LLM, so it is essential to make sure those instructions are well understood. Below are a few best practices for constructing a successful prompt:

1. Specify the Task Clearly and Use Explicit Instructions: Clearly state what you want the model to do and specify the style or format of the response. This reduces ambiguity and ensures that the model provides the desired output.

2. Include Context: Providing relevant context can help the model generate relevant and coherent responses, especially in complex queries.

3. Ask Specific Questions and Limit Open-Ended Inputs: Formulate precise questions to elicit clear and focused responses. Narrow the scope to avoid broad or vague answers.

4. Break Down Complex Tasks: For complex or multi-step tasks, consider breaking them into simpler, sequential prompts.

5. Use Examples to Guide Format: Offering a sample response can set expectations for the type and style of answer you anticipate.

Note that if you have multiple examples to illustrate how you would like the LLM to respond, you can use the Training Examples feature to provide these in a structured way.

6. Provide Constraints When Necessary: Introducing constraints like word limits or particular formats can help control the output.

Sample Prompt:
Assign the interaction a score from 1-5 based on the performance of the agent and provide a one sentence rationale for your score. Evaluate based on the following criteria:

• Professionalism: Was the agent friendly, clear, and professional?
• Problem-Solving: Did the agent resolve the issue efficiently?
• Overall Experience: Was the customer satisfied, and was the call handled well?

Output Format: Text

Sample Result:
Score: 5
Rationale: The agent was courteous, provided clear explanations, & efficiently resolved the issue.

Sample Prompt:
Identify the primary intent of the customer during this interaction. Categorize the intent into one of the following predefined categories: Technical Support, Billing Inquiry, Product Feedback, General Inquiry.

Output Format: Text

Sample Result:
Billing Inquiry

Sample Prompt:
Identify a competitor mentioned in the conversation along with the product or service they offer and the reason for comparison.

Output Format: Text

Sample Result:
Owl Shoes - MaxH Running Shoes: Customer is comparing comfort and durability.

Sample Prompt:
Extract the customer's order number mentioned either by the customer or agent in the interaction. Use numerals for the numbers included in the order number, and capitalize the letters. Also, add a dash in between the first three and final four characters.

Output Format: Text

Sample Result:
180-A84S

Sample Prompt:
Review the transcript and evaluate whether the agent adhered to the required script. Check for the presence of key statements:

• Greeting & Identification (25%) – Did the agent introduce themselves and the company?
• Disclosure Statement (25%) – Did the agent provide the mandatory disclosure?
• Resolution Steps (30%) – Did the agent follow the prescribed resolution process?
• Closing & Next Steps (20%) – Did the agent summarize the resolution and offer further assistance?

Also provide a concise score reason explaining your assigned score.

Output Format: JSON

JSON Schema:

1
{
2
    "type": "object",
3
    "properties": {
4
        "greeting": { "type": "boolean" },
5
        "disclosure_statements": { "type": "boolean" },
6
        "resolution_steps": { "type": "boolean" },
7
        "closing_next_steps": { "type": "boolean" },
8
        "score": { "type": "string" },
9
        "score_reason": { "type": "string" }
10
    }
11
}

Sample Result:

1
{
2
    "greeting": true,
3
    "disclosure_statements": false,
4
    "resolution_steps": true,
5
    "closing_next_steps": true,
6
    "score": "75%",
7
    "score_reason": "Partial compliance. Missing required disclosure."
8
}

Sample Prompt:
Detect strong buying signals in sales conversations, such as pricing inquiries, purchase timeline discussions, and objections that suggest high interest. Classify the lead's likelihood to close.

Output Format: JSON

JSON Schema:

1
{
2
    "type": "object",
3
    "properties": {
4
        "buying_signals": { "type": "string" },
5
        "confidence_score": { "type": "number" },
6
        "lead_category": { "type": "string" }
7
    }
8
}

Sample Result:

1
  {
2
    "buying_signal": "Asked about annual pricing discounts & contract cancellation terms",
3
    "confidence_score": 0.9,
4
    "lead_category": "Hot"
5
  }

Sample Prompt:
Use the following parameters to evaluate the phone call between the agent and the customer for inflammatory content:

• Hostile Language: Did the agent or customer use any language that could be considered aggressive, rude, or hostile? (e.g., yelling, insults, threats, etc.) (Yes/No)
• Personal Attacks: Was there any instance of personal attacks, insults, or discriminatory remarks made by either party? (Yes/No)
• Offensive Language: Did the agent or customer use profanity or offensive language during the conversation? (Yes/No)*

Output Format: JSON

JSON Schema:

1
{
2
    "type": "object",
3
    "properties": {
4
        "aggressive_hostile_language": { "type": "string", "enum": ["yes", "no"] },
5
        "personal_attacks": { "type": "string", "enum": ["yes", "no"] },
6
        "profanity_offensive_language": { "type": "string", "enum": ["yes", "no"] }
7
    }
8
}

Sample Result:

1
{
2
  "aggressive_hostile_language": "no",
3
  "personal_attacks": "no",
4
  "profanity_offensive_language": "yes" 
5
}

Sample Prompt:
Summarize sales conversations in a custom format designed for sales workflows, using the BANT framework:

• Budget: Does the customer have a budget allocated for the solution?
• Authority: Is the customer the decision-maker?
• Need: What problem is the customer trying to solve?
• Timeline: When does the customer need the solution in place?

Output Format: JSON

JSON Schema:

1
{
2
    "type": "object",
3
    "properties": {
4
        "budget": { "type": "string" },
5
        "authority": { "type": "string" },
6
        "need": { "type": "string" },
7
        "timeline": { "type": "string" }
8
    }
9
}

Sample Result:

1
{
2
  "budget": "$50,000 annually, subject to change based on features.",
3
  "authority": "Customer is leading the evaluation, but the CTO has final approval.",
4
  "need": "Improve customer support by managing inbound calls and messages more effectively.",
5
  "timeline": "Implementation needed by Q1 next year."
6
}

Like all Language Operators, Generative Custom Operators will execute immediately following successful creation of a Transcript. Operator results can be retrieved in two ways:

1. In Console
2. Using the REST API

Generative Custom Operator Results are visible on the Transcript Viewer in Console.

To view Operator Results in Console:

• Visit the Transcripts page
• Select your Transcript where the Generative Custom Operator has run

On the Transcript Viewer, you will see the Language Operator Results on the right pane alongside the transcript text. You will see the Friendly Name of your Generative Custom Operator along with the Operator Resuts.

Generative Custom Operator results can be retrieved using the Transcript Operator Results API. This is often preferrable for consuming Operator Results in downstream systems.

In the response to the Operator Results API request, you will receive an array of operator_results. The operator_results property contains an array of results of the analyses from the Voice Intelligence Service's attached Language Operators.

The format of the Generative Custom Operator Result object will depend on whether you created a Generative or GenerativeJSON Operator type.

The results from the LLM will be returned in the text_generation_results object. Specifically, text_generation_results.result will contain the generated text response from the LLM.

Below is a simplified example of the Operator Result object for a Generative Custom Operator:

1
{
2
    "name": "GenerativeOperator",
3
    "url": "https://intelligence.twilio.com/v2/Transcripts/GTXXX/OperatorResults/LYXXX",
4
    "operator_sid": "LYXXX",
5
    "transcript_sid": "GTXXX",
6
    "operator_type": "text-generation",
7
    "text_generation_results": {
8
        "result": "three bedroom apartment",
9
        "format": "text"
10
    }
11
}

Importantly, this object will be returned as one of several operator_results objects in the response to the Operator Results API request.

The full API request/response will use the following format:

GET https://intelligence.twilio.com/v2/Transcripts/GTXXX/OperatorResults/LYXXX

Response:

1
{
2
  "operator_results": [
3
    {
4
        "name": "GenerativeOperator",
5
        "url": "https://intelligence.twilio.com/v2/Transcripts/GTXXX/OperatorResults/LYXXX",
6
        "operator_sid": "LYXXX",
7
        "transcript_sid": "GTXXX",
8
        "operator_type": "text-generation",
9
        "text_generation_results": {
10
            "result": "three bedroom apartment",
11
            "format": "text"
12
        }
13
    }
14
  ],
15
  "meta": {
16
    "page": 0,
17
    "page_size": 50,
18
    "first_page_url": "https://intelligence.twilio.com/v2/Transcripts/GTXXX/OperatorResults?PageSize=50&Page=0",
19
    "previous_page_url": null,
20
    "url": "https://intelligence.twilio.com/v2/Transcripts/GTXXX/OperatorResults?PageSize=50&Page=0",
21
    "next_page_url": null,
22
    "key": "operator_results"
23
  }
24
}

The operator_results object for a GenerativeJSON Operator type will look slightly different, and return the Operator Result in accordance with the JSON schema provided during Operator creation.

Specifically, the JSON will be returned as the json_results object.

Below is a simplified example of the Operator Result object for a GenerativeJSON Custom Operator:

1
{
2
    "name": "GenerativeJSONOperator",
3
    "url": "https://intelligence.twilio.com/v2/Transcripts/GTXXX/OperatorResults/LYXXX",
4
    "json_results": {
5
        "age": 30,
6
        "favorite_number": 7,
7
        "hair_color": "blonde"
8
    },
9
    "operator_sid": "LYXXX",
10
    "transcript_sid": "GTXXX",
11
    "operator_type": "json"
12
}

Importantly, this object will be returned as one of several operator_results objects in the response to the Operator Results API request.

The full API request/response will use the following format:

GET https://intelligence.twilio.com/v2/Transcripts/GTXXX/OperatorResults/LYXXX

Response:

1
{
2
  "operator_results": [
3
    {
4
        "name": "GenerativeJSONOperator",
5
        "url": "https://intelligence.twilio.com/v2/Transcripts/GTXXX/OperatorResults/LYXXX",
6
        "json_results": {
7
            "age": 30,
8
            "favorite_number": 7,
9
            "hair_color": "blonde"
10
        },
11
        "operator_sid": "LYXXX",
12
        "transcript_sid": "GTXXX",
13
        "operator_type": "json"
14
    }
15
  ],
16
  "meta": {
17
    "page": 0,
18
    "page_size": 50,
19
    "first_page_url": "https://intelligence.twilio.com/v2/Transcripts/GTXXX/OperatorResults?PageSize=50&Page=0",
20
    "previous_page_url": null,
21
    "url": "https://intelligence.twilio.com/v2/Transcripts/GTXXX/OperatorResults?PageSize=50&Page=0",
22
    "next_page_url": null,
23
    "key": "operator_results"
24
  }
25
}

For customers with PII redaction enabled on their Voice Intelligence Service, it isn't currently possible to view Generative Custom Operator results with PII redacted. Fetching Generative Custom Operator Results for a redacted transcript will return an entirely redacted value by default.

Below is an example of what the Operator Results API response will look like if PII redation is enabled by default:

1
"text_generation_results": {
2
    "result": "[Redacted: GenerativeOperatorFriendlyName]",
3
    "format": "text"
4
}

To view Generative Custom Operator results for redacted transcript it is necessary to set the Redacted Query Parameter equal to false to return the original unredacted Operator Results.

Example: GET https://intelligence.twilio.com/v2/Transcripts/{TranscriptSid}/OperatorResults/?Redacted=false

For more information on fetching Generative Custom Operator Results, please see the Transcript Operator Results API documentation

Unlike the previous generation of language operators, Generative Custom Operators will be billed based on characters included in the input to and output from the LLM.

• Input: includes all information sent to the LLM to generate a response
• Output: The actual results returned by the LLM

Please see Twilio Voice Pricing for the amounts billed per 1k characters.

There are three main components of the billable input character count:

1. Transcript: The transcription of the call itself required for the LLM to perform conversational analysis. We estimate roughly 800-1000 characters per 1 minute of call recording time.
2. User prompt: The instructions to the LLM that you provide. This is inclusive of the prompt itself (spaces included), plus training examples and the JSON output format, if using those advanced settings.
3. System prompt: Additional metadata provided by Twilio required for the LLM to execute successfully. We estimate this to be between 100-200 characters per operator execution, on average.

To estimate the number of input characters per operator execution, simply add your average transcript character count to user and system prompt characters. A simple example with conservative figures:

• 2m transcript = 2000 characters
• User prompt = 630 characters
• System prompt = 200 characters
• Total = 2830 characters

Output character count is straightforward – it is limited to the data returned by the LLM when your Generative Custom Operator runs. The character count of the output will be determined based on a combination of:

• The instructions you provide in the prompt
• The output format you specify (text or JSON)

The best way to estimate output character count is either to preview your operator results during creation, or look at actual Operator Results.