Build to scale: queueing and latency on Twilio

Delivering messages at scale is a top-of-mind consideration when building applications for high-traffic events, such as Black Friday. Therefore, we’ve created a guide of best practices that you can follow to ensure your applications with Twilio scale easily.

Example: Black Friday sales

Imagine you have a list of 1,000 products that go on sale at 11:00 AM the morning of Black Friday. Now, you have one million eager end users who want to be notified of your Black Friday deals as soon as they’re live. You're planning to send out a blast of promotional SMS messages to that entire group at 10:00 AM Black Friday morning to let them know the deals will be live in an hour.

You also know the best products run out within a few minutes. To protect against automated bots and to ensure that your potential audience has an equitable experience, you need to incorporate an OTP code verification step as part of the checkout flow. In this case, you require two messages to safely and securely facilitate the sale.

Based on your use case, on Black Friday, you have two separate use cases for Twilio Messaging: one for promotional messages sent an hour before the sales go live and one for OTP verification messages during checkout. For each of these use cases, we recommend creating a Messaging Service with a dedicated Sender Pool and Validity Period setting.

• Message 1, Promotional - The end user gets a marketing link in a message just before your Black Friday sales go live. The link takes them to an application where they have to login or signup.
• Message 2, OTP - Upon signup, the end user must enter a one-time-passcode (OTP) pin for logging in. This message has to arrive immediately, so that the user can purchase before the items run out! For this example, let's conservatively assume that 2% of your audience clicks the link and eventually, 0.1% buy.

Using Messaging Services

You can think of a Twilio Messaging Service as a bundling of both senders (in this case, phone numbers) and settings for a specific messaging use case. When you need to consider messaging at scale, a Messaging Service handles intelligent routing and features like opt-out management for you, across multiple senders. So instead of configuring multiple individual numbers, you can create a Messaging Service and add as many senders to the sender pool as you need. The configuration will apply to all of the numbers in your sender pool.

What happens when you make a message request to a Messaging Service?

When you send a Message with a Messaging Service, the Service queues the API Request before sending it to the specific Sender, then onto the Carrier for eventual delivery to the end user. Instead of a from phone number, you will enter a Messaging Service SID in your API request to Twilio.

A different Messaging Service for each use case

Recall that in our Black Friday example, we plan to send two types of Messages: one promotional, pre-sale message and one OTP verification message.

In this case, you'll want to create two different Messaging Services, one for each use case. Each Messaging Service will have a separate Sender Pool (or group of numbers)

How to build for scale with Twilio Messaging Services

Once you have defined your use cases, you can plan for your anticipated messaging traffic and ensure that the messages are delivered in the necessary timeframe.

1. Forecast your traffic and delivery timeline requirements

For each messaging service, we recommend definining both how many messages you'll need to send (traffic) and the delivery timeframe. For example, in our Black Friday example, the Promotional and OTP use cases have different levels of traffic and time-sensitivity, so we want to queue up our messages differently.

1. Promotional: Let's say you need to send 1,000,000 messages to your audience through the special day. You may queue up a million message requests on your Twilio Promotional Messaging Service.
2. OTP: Let's say 2% customers clicking on the link to sign up or to login and purchase items at exactly 5 PM when sales go live. OTP messages need to be delivered urgently, let’s assume within 180 seconds.

2. Calculate the queue buildup and MPS rate for your application

Calculating your MPS (Messages Per Second) rate is crucial for building for scale. This will determine how big your queue will get and the rate at which messages will be sent.

Twilio queues your messages for 4 hours (14400 seconds) by default, so we will use this for the example calculations.

To calculate your MPS, divide the number of messages to be sent over the queue length. This will give you the required MPS limit to send out your messages. That means we would need to plan for the right Senders in your Messaging Service Sender Pool to be able to send messages at 2x your MPS.

To be safe, Twilio suggests that customers apply a 100% buffer to this MPS requirement, in the event that Telco carriers have capacity issues.

1. Promotional: To calculate our MPS, we divide our 1M promotional messages by 14,400 seconds, about 70 Messages need to be sent per second. In the name of safety, we'll have our the senders in our Messaging Service's Sender Pool send messages at a rate of 140 MPS.
2. OTP: Say you apply a 180-second validity period for 20,000 messages, with that buffer factor of 2x. That would require a 220 MPS limit requirement.

(We'll cover how to meet your various MPS limit requirements by provisioning your Senders in a later step.)

3. Set up Messaging Services for each of your use cases

After you've determine the key factors affecting your messaging for your use case(s), you can create Messaging Services for each one.

1. Promotional: Leave the Validity period to the default (4 hours/14400 seconds).
2. OTP: Set the Validity period to be a short duration, say 2 minutes (120 seconds).

You can create and manage your Messaging Service in the Services section of the Twilio Console or using the REST API using the Service Resource.

4. Provision Senders into your Messaging Service Sender Pools

In a previous step, you calculated your MPS requirements and added a buffer to make sure your messages were sent in the appropriate window of time for your two use cases. To meet these requirements, you'll want to add different Senders to your Messaging Services' Sender Pools.

To adhere to carrier requirements, each Sender (phone number) on Twilio has specific MPS limits. These limits vary by both Sender type and location. For example, Short Codes have a 100 MPS limit. A Long Code in the US has 1 MPS, but a Long Code in the UK could have 10 MPS.

Include backup Long Codes

When a particular type of Short Code Sender cannot deliver messages to a recipient, Twilio Messaging Services will automatically failover and retry messages with Long Codes, so we recommend including some backup Long Code Senders in your Sender Pool.

Now, let's see how this all applies to our Black Friday example use cases:

1. Promotional: For the 140 MPS that we calculated before, you can provision some high-capacity A2P senders like Short Codes or High Throughput Toll Free numbers, and some Long Codes for Fall back.

2. OTP: For the 220 MPS needed, you could also provision some high-capacity A2P senders and some long codes for Fall back. However, turn off the Sticky Sender feature to ensure delivery speeds. Sticky Sender forces our systems to use the same number that we have used in the past to contact your consumer, but this might cause certain numbers to be queued up too heavily. Turning off this feature enables better load balancing for your messages. (You can read more Sticky Sender and other Messaging Service features.)

5. Time your message campaigns

Now that you have Messaging Services with the appropriate configurations and Sender Pool, you can set up your messages to be delivered on time with the Twilio Programmable Messaging API.

1. Promotional: Queue them up in the morning so your messages have plenty of time to get delivered.
2. OTP: Set up your application to automatically send OTP messages as needed.

6. Monitor your Messaging Capacity via the Latency Report on Messaging Insights

You can keep an eye on your messaging application using Twilio's Messaging Insights in the Twilio Console. (Check our our doc on Exploring Messaging Insights.)

Within Messaging Insights, the Latency Insights report will highlight if there are latency issues with any of your Senders, Messaging Services, or Twilio sub-accounts, in near real-time. The Messaging Insights Delivery & Errors report will help you determine the health of delivery rates and identify if there are certain errors you should troubleshoot.

View the Latency Report to see how quickly your messages are being processed by the Twilio platform. If there are issues contributing to increased latency of your messages on Twilio’s platform, they will show up in the charts dedicated to Sender, Messaging Service and Account-level queueing.

When any of these entities--Senders, Messaging Services, or Account-level issues--cause too much latency and large queues, you can see the send rate limits by hovering over the column of interest and clicking the View Limits button. Twilio also provides guidance of what to do next within the modals that appear upon clicking.

You can use the dashboards in Messaging Insights to monitor your messaging throughout the day, as the sales roll in from your smart pre-Black Friday planning.

7. Handling large emergency use cases

Say your billing systems go down and you need to notify all your customers that sales will be delayed for a while. Your normal account limits may not suffice in such large situations. Reach out to Twilio and we may be able to help by adjusting the limits applied to your account, senders, or messaging services.

Handy reference guides for calculating queueing and sender needs

Here’s a quick reference on how you could calculate your queueing / sender needs as described above.

Promotional Use Case Messaging Service

• Messages to Send: 1000000 message segments
• Validity period (default) 14400 seconds
• Approximate MPS needed: 70 messages/second
• Applied Buffer factor: 200%
• Total MPS to plan for: 140 messages/second

* Total MPS Capacity = Quantity * MPS Capacity

OTP Use Case Messaging Service

• Assumed Click Conversion to Login: 2%
• Number of OTP messages: 20,000 message segments
• Validity Period: 180 seconds
• Approximate MPS needed: 110
• Buffer applied 200%
• Total MPS to plan for: 220

* Total MPS Capacity = Quantity * MPS Capacity

8. Gracefully handle errors

Customers sending large volumes of messages may encounter errors such as Queue Overflow, in order to track these errors programmatically use Twilio's Debugger by setting a webhook for these types of errors. If you, for example, get a 30001 error you can slow down your application by implementing backoff and retries.

Where to Next?

This guide walked you through how to set up two Messaging Services for a time-sensitive, high-volume event using Twilio Messages. As you can scale, you can implement the steps and strategies in this guide across multiple Twilio Accounts, each containing one or more Messaging Services as you build different applications. We can’t wait to see what you build.

• Explore Messaging Insights
• Messaging Service Features
• How to Send Messages with Messaging Services