BulkExport API Overview
This feature is not currently available when using Twilio Regions Ireland (IE1) or Australia (AU1). This is currently only supported with the default US1 region. A full list of unsupported products and features with Twilio Regions is documented here.
The BulkExport API provides an efficient mechanism for retrieving all of your activity logs from the Twilio platform on an ongoing basis, or for one-off downloads. Using BulkExport, you can access daily dumps of the previous day’s Messages, Calls, Conferences, or conference Participants, eliminating the need to iterate through the list resource one page at a time to download your records. The BulkExport files are automatically removed after seven days. You can also use BulkExport to generate a set of historical daily dumps over a range of dates.
When a BulkExport completes, it can fire a webhook and/or send an email. This applies to the daily export, if configured, and to custom jobs. In both cases, successful exports will generate a Day resource for each day requested.
BulkExport is useful if you have:
- A data warehouse where you want to store messaging traffic to perform analysis.
- A compliance store where messaging should be retained so that you have a permanent record.
- Requirements to produce data.
Base URL
Control over BulkExports, and the data exported is available under the following URL:
https://bulkexports.twilio.com/v1/Exports/<resource_type>
How BulkExport works
There are four parts to BulkExport:
- A configuration that allows customers of Twilio Edition products to set automatic exports for each day.
- A way to export a specified day’s traffic.
- A means to manage exports in progress.
- A list of exported days with their output files.
ExportConfiguration
The ExportConfiguration resource lets you specify if you have a Twilio Editions product, whether exports will take place automatically every day. This makes getting the data much more simple: you just configure the webhook. This will be called when the day’s file is available, and you can then download and process your data.
Custom Jobs
Jobs lets you choose a range of dates to export data for. One data file will be generated for each day in the range. A parent account and all associated sub accounts can request data for up to a total of 366 days within a UTC calendar day. If making a request from a sub account, the parent account will be used to determine the amount of days being requested within a UTC calendar day. For instance, if you submit requests across several sub-accounts, and the aggregate number of requested days exceeds 366 within a 24-hour period, your request will be rejected. In this instance, you'll need to wait until the next UTC calendar day to make any further requests. The result data for the requested days will be visible using the Day resource. You will receive a callback and/or email for each job that completes. When a custom job completes, it will generate one Day resource for each day in the requested time period.
By default, the system will send bulk export job initiation notifications to account Owners and Admins, so they can keep a tab on the export requests, and flag/delete unwanted requests. Admins will be able to unsubscribe to these notifications from the notification center on the console.
Jobs
Get a list of jobs in progress or which have recently completed. Completed jobs will appear on the job list for seven days. You can delete jobs before they are complete.
Exported days
The Days resource lets you examine the list of exported days available, either because they have been already generated by an export job or an automatic daily export. Each day file is named with the ISO 8601 date, and days are listed in order of generation.
There is no way to delete completed Day files, but they will be deleted automatically seven days after their creation.
Subresources
Resource | Description |
ExportConfiguration | Configure export webhooks. Twilio Enterprise Edition customers can also set daily exports. |
ExportCustomJob | Create new jobs to export data, and list current jobs and their status. Running a job will create the corresponding Days. |
Jobs | View the status of current jobs, and delete incomplete custom jobs. |
Days | List and fetch the exported message data for given days. |
Using BulkExport
There are two ways to use BulkExport:
- If you have the Twilio Enterprise Edition, you can enable the daily export using the ExportConfiguration resource. This will generate one data file per day for every resource enabled. The system typically tries to create a data file based on the state of the data at that point, but may be delayed occasionally.
- Create custom jobs using the ExportCustomJob resource.
You can then use the Day resource to list all completed exports by day and to download the message data for any of those days.
Messages BulkExport file format
Day files are stored and returned as compressed (Gzip) JSON files with one record per line. Records are similar to the Message resource. The differences are:
- Timestamps are in UTC in ISO 8601 format.
- Price data may not be present in every message in BulkExport.
- The following fields are not present:
price_units
api_version
error_message
(buterror_code
is)uri
subresource_uris
(containing the media URLs)
Calls BulkExport file format
Day files are stored and returned as compressed (Gzip) JSON files with one record per line. Records are similar to the Calls resource. The differences are:
- Timestamps are in UTC in ISO 8601 format.
- Price data may not be present in every message in BulkExport.
- The following fields are not present:
uri
subresource_uris
NOTE: Using the DELETE
method on Archived Calls API resources does not also perform a DELETE
operation on those resources under the Voice API. For resources less than 13 months old, deleting calls via the Voice API will also result in them being deleted under the Bulk Export API. Resources older than 13 months can only be deleted via BulkExport API.
Conferences BulkExport file format
Day files are stored and returned as compressed (Gzip) JSON files with one record per line. Records are similar to the Conferences resource. The differences are:
- Timestamps are in UTC in ISO 8601 format.
- The following fields are not present:
api_version
uri
subresource_uris
Participants BulkExport file format
Day files are stored and returned as compressed (Gzip) JSON files with one record per line. Records are similar to the Participants resource. The differences are:
- Timestamps are in UTC in ISO 8601 format.
- The following fields are not present:
uri
label
Need some help?
We all do sometimes; code is hard. Get help now from our support team, or lean on the wisdom of the crowd by visiting Twilio's Stack Overflow Collective or browsing the Twilio tag on Stack Overflow.