virtual-background

Virtual Background Project Introduction

The Virtual Background project is an innovative browser-based application designed to add virtual backgrounds to live video streams. This feature enriches video calls and presentations by seamlessly integrating artificial backgrounds in real-time, offering users increased privacy and a touch of creativity during virtual interactions. Interested individuals can experience the functionality live by visiting this demo.

Implementation Details

This project utilizes three state-of-the-art machine learning (ML) pre-trained segmentation models:

• BodyPix: This model provides tools to segment the human body from the background. However, its built-in methods like toMask and drawMask are not used in the demo for optimal performance, as they are not fully optimized for simpler background tasks. Instead, the project uses techniques with Canvas 2D for blending layers.

• MediaPipe Meet Segmentation: Available as a TensorFlow Lite model, it is particularly optimized for background segmentation. The project uses a WebAssembly tool to deliver high performance by leveraging TFLite with XNNPACK and SIMD, closely following Google’s approach as described in their blog.

• ML Kit Selfie Segmentation: This model works similarly to MediaPipe's approach and appears to be derived from the same foundational Keras model. It offers high-quality segmentation, although slightly less performant than the MediaPipe option due to higher input resolution requirements.

Performance

The application performs efficiently on various devices. On a Pixel 3 smartphone using Chrome, different configurations present varying frames per second (FPS), indicating how swiftly the application processes the video stream:

• MediaPipe and ML Kit models offer better FPS with the use of WebAssembly SIMD and WebGL 2 configurations, maximizing resource utilization and enhancing visual output quality.
• BodyPix, while functional, delivers lower FPS rates due to its reliance on WebGL without optimizations suited to mobile platforms.

Possible Improvements

To elevate the performance and visual appeal further, the project suggests:

• Utilizing the alpha channel to minimize texture fetches.
• Pre-blurring background images for better composite results.
• Optimizing shaders for faster processing during rendering operations.
• Exploring multithreading to build TFLite with enhanced performance capabilities.

Related Work

The project builds on cutting-edge technologies, combining the strengths of WebAssembly, WebGL, and TensorFlow's segmentation models to deliver a user-friendly experience in real-time video streaming environments. The details of these components can be found in their respective repositories and documentation, such as the BodyPix repository.

Running Locally

To engage with the Virtual Background project on a personal device:

1. Use the command yarn start to run the application in development mode.
2. Open the browser at http://localhost:3000 to view and interact with the application.
3. Use yarn test for checking any code issues and yarn build to prepare the app for production deployment.
4. Building the TensorFlow Lite tool requires Docker and specific commands such as yarn build:tflite to create the necessary WASM inference tools.

This project is a robust gateway to integrating virtual backgrounds, blending technical sophistication with practical usage in live video contexts.