Adding Virtual Backgrounds

Twilio Video supports background blur and replacement on the web using the optional Video Processors JavaScript library. The Video Processors library is a collection of video processing tools that can be used with the Twilio Video JavaScript SDK to apply transformations and filters, such as background blurring and virtual backgrounds, to a VideoTrack. You can also use the tools to create your own transformations or filters.

See a live demo with blurred backgrounds and virtual backgrounds in the browser using the Video Processors tools here.

View the Video Processors demo application and run the example application locally.

The recent changes to the library are documented here

• Twilio Video JavaScript SDK (v2.29+)
• Node.js (v18+)
• NPM (v10+, comes installed with newer Node versions)

You can install the Video Processors library using Node Package Manager (npm):

npm install @twilio/video-processors --save

Using this method, you can import twilio-video-processors like so:

import * as VideoProcessors from '@twilio/video-processors';

After installing with npm, you can also copy twilio-video-processors.js from the dist/build folder and include it directly in your web app using a <script> tag:

<script src="https://my-server-path/twilio-video-processors.js"></script>

Using this method, twilio-video-processors.js will set a browser global:

const VideoProcessor = Twilio.VideoProcessors;

Add the video processor to the Video track.

1
videoTrack.addProcessor(VideoProcessor, {
2
  inputFrameBufferType: 'videoframe',
3
  outputFrameBufferContextType: 'bitmaprenderer'
4
});

Video frame processing is handled by web workers on all major browsers preventing the main thread from blocking. Support for cross-domain worker hosting is now available. Below is an example of a cross-domain configuration:

1
import { GaussianBlurBackgroundProcessor } from '@twilio/video-processors';
2

3
/* Application is running at https://example.com/app */
4

5
const processor = new GaussianBlurBackgroundProcessor({
6
  assetsPath: "https://example.net/path/to/assets"
7
});

Note that this requires the Access-Control-Allow-Origin headers to properly point to your application domain.

View the latest Video Processors JavaScript library documentation here.

Documentation for version 2.x Video Processors JavaScript library is available here.

The following table lists the browsers and operating systems supported by the Video Processors JavaScript library.

The Video Processors library is CPU intensive as it processes each video frame individually. The library uses Web Assembly and will offload some video processing to the GPU; however, there is a minimum hardware requirement to ensure that the device isn't overpowered and has resources to support the ongoing video call.

The following is the minimum hardware specification:

• CPU: Intel i5-7200
• GPU: Intel HD Graphics 620
• RAM: 8GB
• OS: Windows 10 Pro

Example: HP ProBook 450 G4

You can use virtual backgrounds for the Video Android SDK or Video iOS SDK by using the optional plugin libraries. The plugins support both background replacement and background blurring. The plugin applies a segmentation filter to the incoming video frames from the camera to either blur or replace the background. The plugin preserves pixels associated with the head, face, and shoulders.

In this guide, you'll learn how to use virtual backgrounds with the Video Android SDK and Video iOS SDK.

The virtual background functionality for Android is available in the VirtualBackgroundProcessor plugin library. This includes both VirtualBackgroundVideoFrameProcessor and BlurBackgroundVideoFrameProcessor background image filters.

The following code snippet shows how to apply a virtual background frame processor to a VideoCapturer instance to modify the incoming frames from the camera.

1
VideoCapturer capturer = Camera2Capturer(context, "myCameraId", new BlurBackgroundVideoFrameProcessor(context, 15), null);
2
localVideoTrack = LocalVideoTrack.create(this, true, capturer, "myAudioTrackName");

Note: When using the VirtualBackgroundProcessor plugins, you should include the following code in your proguard-rules.pro file:

1
 -keep class com.google.mlkit.common.** { *; }
2
 -keep class com.google.mlkit.vision.** { *; }

The virtual background functionality for iOS is available in the TwilioVirtualBackgroundProcessors plugin library. With this library, you can extend the TVIBackgroundProcessor delegate to implement a background processor or you can use the TVIDefaultBackgroundProcessor API to perform background processing. To use the TVIDefaultBackgroundProcessor, pass a TVIDefaultBackgroundProcessor instance when creating a TVICameraSource object.

Note: TVIDefaultBackgroundProcessor is only supported on iOS 17 or later. Loading this library in apps with a lower deployment target will result in unexpected behavior.

TVICameraSource has an initializer API that accepts an instance of TVIDefaultBackgroundProcessor or a custom implementation of TVIBackgroundProcessor protocol.

1
- (nullable instancetype)initWithOptions:(nonnull TVICameraSourceOptions *)options
2
                                delegate:(nullable id<TVICameraSourceDelegate>)delegate
3
             backgroundProcessorDelegate:(nonnull id<TVIBackgroundProcessor>)backgroundProcessorDelegate

The following example shows how to use the API:

1
    var camera: CameraSource?
2
    var backgroundProcessor: DefaultBackgroundProcessor?
3
    var backgroundImage: UIImage?
4
    
5
    backgroundProcessor = DefaultBackgroundProcessor(backgroundImage: image)
6
    // Initialize CameraSource with the background processor
7
    camera = CameraSource(options: options, delegate: self, backgroundProcessorDelegate: backgroundProcessor!)

You can pause and resume virtual background processing by toggling the pauseProcessing property of TVIDefaultBackgroundProcessor.

    self.backgroundProcessor?.pauseProcessing = true

You can set the backgroundImage or blurFilterRadius property values while the virtual background processor is active. Note that these two properties are mutually exclusive. For example:

• If you set an image for backgroundImage, the value of blurFilterRadius will be set to 0.0.
• If you set blurFilterRadius, the backgroundImage value will be set to nil.
• If you set backgroundImage to nil and blurFilterRadius to 0.0, the video frame will be delivered to the video sink unprocessed.

The TwilioVirtualBackgroundProcessors plugin library is available in both static and dynamic framework formats. You must use the same framework type (either static or dynamic) for both the Twilio Video iOS SDK and the TwilioVirtualBackgroundProcessors plugin library. Using different framework types can cause build and runtime problems.