Twilio Changelog | Nov. 12, 2024
Twilio Virtual Agent with Dialogflow CX now supports Generative AI capabilities and Custom Voice Prompts
TLDR; Google’s latest Vertex/Gemini Generative AI models and capabilities are now supported in Twilio’s Virtual Agent with Google Dialogflow CX one-click connector integration – plus Custom Voice Prompts
Twilio’s <Connect><Virtual Agent> connector, Studio Widget and TwiML now support use of Google’s latest Generative AI capabilities and models – four methods of incrementally wading into the world of Generative bots for voice self-service automation.:
Generative Playbooks
Generative Fallback
Generators
Data Store Agents
These enhancements are in addition to Dialogflow’s traditional Predictive AI-based flows, which are deterministic, and based on detected natural language spoken “intents” and caller sentiment.
Customers can also now upload and use their own Custom Voices with which to prompt customers in Dialogflow virtual agents, so they may preserve common or unique brand identity with or versus other (non-automated) voice experiences they deliver to their customers.
How can you take advantage of these new Generative AI and Custom Voice Prompt capabilities of Dialogflow?
Use of these features will be billed by Twilio for Dialogflow services, in the same manner as with regular (predictive/non-generative) Dialogflow usage. See Twilio’s pricing page for details.
For Custom Voices, we've also added an additional configuration option for specifying the custom synthesis voice model for Dialogflow in TwiML It can be provided in the form <Config name="voiceModel" value="projects/{project}/locations/{location}/models/{model}"/> nested inside <Connect><VirtualAgent> similar to other configuration parameters. As mentioned in the Dialogflow docs, you will also need to grant an additional IAM permission to Twilio's service account used for the integration (virtualagent-ccai-prod@dialogflow-prod-env.iam.gserviceaccount.com) for the model:
Authorization requires the following IAM permission on the specified resource model:
automl.models.predict
More info on how to use both is available in the Twilio and Google Dialogflow documentation links below.
Customer benefits
Builders looking to deliver natural language, conversational self-service automation (IVR or Virtual Agent) caller experiences who want to start using Generative AI in targeted scenarios – without turning their entire caller experience over to GenAI prompt engineers at once – can begin by just using Generators or Generative fallback to add a more conversational quality where predictive AI may feel more scripted, and expand GenAI’s use incrementally in scenarios most suited to natural conversation from there, without converting their entire virtual agent to GenAI in one fell swoop. Combined with Custom Voices prompting, Customers can deliver an on-brand, natural-sounding, self-service automation experience second to none, refined/tuned and personalized, at production scale, with Twilio’s tools.
Learn More
https://cloud.google.com/dialogflow/cx/docs/generative-deterministic
https://www.twilio.com/docs/voice/twiml/connect/virtualagent/virtualagent-dialogflow-cx#dynamic-connector-configuration
https://cloud.google.com/dialogflow/cx/docs/concept/agent-settings#voice-selection
https://www.twilio.com/docs/voice/virtual-agent/dialogflow-cx-onboarding