Managing codecs

The codecs that Twilio Video Client SDKs support are platform dependent. In the JavaScript SDK, browser vendors supply the codec implementations, while on mobile SDKs the device hardware determines codec support. Modern browsers and recent Twilio Video SDKs support the VP8, VP9, and H.264 Video codecs.

Twilio Video SDKs support the following video codecs.

For details, see the browser support table.

[1]: VP9 is supported only in Peer-to-peer Video Rooms.

[2]: Twilio Video for iOS relies on hardware support for H.264. Hardware encode/decode is supported on all iOS devices.

[3]: Twilio Video for Android relies on hardware support for H.264, which depends on device capabilities. Reference the following example to check if the device supports H.264.

The following example shows how to check if a device supports the H.264 codec (Video Android 6.x+):

1

2
HardwareVideoEncoderFactory hardwareVideoEncoderFactory =
3
        new HardwareVideoEncoderFactory(null, true, true);
4
HardwareVideoDecoderFactory hardwareVideoDecoderFactory =
5
        new HardwareVideoDecoderFactory(null);
6

7
boolean h264EncoderSupported = false;
8
for (VideoCodecInfo videoCodecInfo : hardwareVideoEncoderFactory.getSupportedCodecs()) {
9
    if (videoCodecInfo.name.equalsIgnoreCase("h264")) {
10
        h264EncoderSupported = true;
11
        break;
12
    }
13
}
14
boolean h264DecoderSupported = false;
15
for (VideoCodecInfo videoCodecInfo : hardwareVideoDecoderFactory.getSupportedCodecs()) {
16
    if (videoCodecInfo.name.equalsIgnoreCase("h264")) {
17
        h264DecoderSupported = true;
18
        break;
19
    }
20
}
21

22
boolean isH264Supported =  h264EncoderSupported && h264DecoderSupported;

Twilio Video SDKs support the following audio codecs.

If your use case averages three or more participants in a Video Room then Twilio recommends using VP8 as the preferred video codec in the connect options and enabling the Adaptive Simulcast feature. Adaptive Simulcast ensures that any network or device issues with one participant do not impact the experience of other participants.

For use cases where the vast majority of Rooms have only two participants, Twilio recommends letting the underlying platform determine the best video codec by not specifying a preferred video codec and leaving Adaptive Simulcast disabled.

Opus is the default and recommended audio codec because it's supported by all modern browsers and the Video Twilio SDKs.

Before establishing a multimedia communication, the involved parties need to agree on the codecs to be used. Twilio manages this through an automatic codec negotiation that imposes some interoperability restrictions.

In a Video Room, an SFU (Selective Forwarding Unit) media server mediates among clients. If client A wants to send media to clients B and C, then A sends only one media stream to the SFU that, in turn, forwards it to B and C. The codec negotiation is based on the following principles:

• For inbound tracks (i.e. tracks getting into the SFU), the SFU tries to accept the codec preferred by the client as long as this codec is supported by the SFU.
• For outbound tracks (i.e. tracks getting out the SFU), the tracks are only offered in the codec in which they are being received.

These have interoperability implications. First because the only codecs allowed are the ones supported by the SFU. The following tables summarize them:

Video Room SFU infrastructure supports the following video codecs.

Video Room SFU infrastructure supports the following audio codecs.

Second, because the codecs negotiated in a call might limit the interoperability with clients joining later. As a result, a set of clients supporting a common codec doesn't guarantee that they'll be able to communicate. For example, participants A and B are using Chrome and Firefox respectively and start communicating using VP8. When participant C joins later with a legacy Safari < 12.1 browser it will not be able to receive video track from A and B because legacy Safari < 12.1 supports only H.264. Even though both A and B could communicate using H.264, when they connected to the SFU they negotiated their default as VP8. The following image illustrates the example.

Expand image

You can override default codec selection using codec preferences. The following SDK versions support codec preferences:

• JavaScript SDK v1.3+
• Android SDK v2.0+
• iOS SDK v2.0+

Codec preferences are based on the following principles:

• Allows selecting both the preferred audio codecs and the preferred video codecs in which media tracks are published by a client SDK.
• Selected codecs are provided as an ordered list of codec specifications, so that codecs specified first have higher preference.
• Only codecs supported at a given SDK can be specified as preferred.
• Codec Preferences are set when a participant connects to a room as part of the ConnectOptions.
• A participant selecting a preferred codec does not guarantee that the SDK uses the codec. The preference just means that the codec shall have higher priority among the list of possible codecs to use. The chosen codec might depend on other factors, such as the codecs supported by the other parties of the communication.

The following examples show how a Participant can connect to a room preferring iSAC as the audio codec and H.264 as the video codec.

Select preferred codecs in Android SDK (v2.0+):

1
// Android Java
2

3
// Prefer H264 if it is hardware available for encoding and decoding
4
VideoCodec videoCodec = isH264Supported ? (new H264Codec()) : (new Vp8Codec());
5
ConnectOptions connectOptions = new ConnectOptions.Builder(token)
6
    .preferAudioCodecs(Collections.singletonList(new IsacCodec()))
7
    .preferVideoCodecs(Collections.singletonList(videoCodec)
8
    .build();
9

10
Room room = Video.connect(context, connectOptions, listener);

Note that checking for H.264 hardware support changed from v5.x to v6.x. See Codecs supported by Twilio Video SDKs to learn how to use the correct syntax.

Selecting preferred codecs in iOS SDK (required v2.0+)**

1
// iOS Swift
2
let options = TVIConnectOptions.init(token: accessToken block: {(builder: TVIConnectOptionsBuilder) -> Void in
3
    builder.preferredAudioCodecs = [ TVIIsacCodec() ]
4
    builder.preferredVideoCodecs = [ TVIH264Codec() ]
5
}
6

7
var room = TwilioVideo.connect(with: options delegate: self)

1
// iOS Objective-C
2
TVIConnectOptions *options = [TVIConnectOptions optionsWithToken:self.accessToken
3
                                block:^(TVIConnectOptionsBuilder * _Nonnull builder) {
4
    builder.preferredAudioCodecs = @[ [TVIIsacCodec new] ];
5
    builder.preferredVideoCodecs = @[ [TVIH264Codec new] ];
6
}];
7

8
TVIRoom *room = [TwilioVideo connectWithOptions:options delegate:self];

For further information on how Codec Preferences work, please check the corresponding SDK reference documentation regarding the connect primitive and its ConnectOptions parameter.

In Twilio Video, a Room uses multi-codec capabilities when different participants send their video or audio using different codecs. Multi-codecs can optimize battery life, because many vendors provide hardware acceleration for specific codec suites. For example, all iOS devices have H.264 hardware acceleration, and many modern Android devices provide hardware acceleration for both H.264 and VP8. If client devices have different hardware support, Codec Preferences make it possible to send media using the codec supported by the local device and to receive media using the codec supported by the remote participant device.

For example, three clients A - Chrome, B - Firefox, and C - iOS smartphone negotiate VP8 as the video codec by default. However, you might prefer to use H.264 for client C to optimize the battery life of the iOS smartphone. You can set H.264 as C's preferred codec. The following code illustrate this example:

Participant A codec preferences:

1
// We don't really need this: VP8 is Chrome's default preferred video codec
2
const room = await connect(token, {
3
    preferredVideoCodecs: ['VP8']
4
});

Participant B codec preferences

1
// We don't really need this: VP8 is Firefox's default preferred video codec
2
const room = await connect(token, {
3
    preferredVideoCodecs: ['VP8']
4
});

Participant C codec preferences:

1
// iOS Swift
2
let options = TVIConnectOptions.init(token: accessToken block: {(builder: TVIConnectOptionsBuilder) -> Void in
3
      builder.preferredVideoCodecs = [ TVIH264Codec() ]
4
}
5
var room = TwilioVideo.connect(with: options delegate: self)

If A, B, and C connect to a Room, the topology is the following:

Expand image

• The quality of H.264 support on Android devices varies and there can be device specific encode and decode issues
• Multi-codec support for video is currently limited to H.264 and VP8.
• Multi-codec support for audio is currently limited to Opus and PCMU.
• Support for H.264 video is currently limited to the Constrained Baseline profile at level 3.1. This means that the maximum resolution of a video track is 1280x720 at 30 fps.
• The use of multi-codecs in a Room with recording activated causes recordings to be also multi-codec. This means that a recording keeps the codec of the track originating it.