Project Introduction: FasterLivePortrait
The FasterLivePortrait project is an innovative endeavor that aims to bring static portraits to life in real time. This project is built upon the foundations of the original LivePortrait repository, which was generously made open-source by its creators. The new improvements and features have made it considerably faster and more versatile, allowing for broader usage and easier deployment.
Key Features and Advancements
-
Real-Time Rendering: Thanks to the power of TensorRT, FasterLivePortrait can execute in real-time on an RTX 3090 GPU, achieving over 30 frames per second. This ensures smooth and seamless animation of images, far beyond just model inference speed, covering rendering as a whole.
-
Cross-Platform Deployment: By converting the LivePortrait model into an Onnx model, the project can now run across different platforms using onnxruntime-gpu with a speed of approximately 12 frames per second. This enhancement facilitates broader application and integration across devices.
-
User-friendly Application: The project smoothly integrates with the native Gradio app, enhancing speed and allowing for simultaneous inference on multiple faces and even animal models. This makes it an excellent tool for creative professionals who need to animate various subjects in their projects.
Recent Updates and Improvements
The project has gone through several improvements, with key updates including:
-
Enhanced Speed and Bug Fixes: Optimizations using torchgeometry and CUDA have significantly boosted performance. Bugs, particularly those pertaining to Xpose operations affecting certain GPUs, have been resolved.
-
Animal Model Support: For those concerned about copyright, FasterLivePortrait now supports animal models, offering more flexibility and creative freedom in animations.
-
MediaPipe Model Integration: Users can replace the InsightFace model with MediaPipe, ensuring diverse application options without legal encumbrances.
-
Simplified Windows Usage: There is a Windows integration package available, requiring no prior installation. This ease-of-use feature allows users to run the application with a single click, enhancing accessibility for Windows users.
-
MacOS Compatibility: MacOS is supported too, especially on M1/M2 chips, although it’s noted that performance may still be limited.
Setup and Usage
FasterLivePortrait can be set up using either Docker for an optimized environment or a Python virtual environment. Specific instructions guide users on installing necessary components such as ffmpeg, Onnxruntime-gpu, and TensorRT depending on their platform of choice.
Various Inference Options
-
Onnxruntime Inference: This allows for CPU-based inference, though it is slower compared to GPU alternatives. Steps are detailed for those interested in setting it up from source.
-
TensorRT Inference: Offers a highly efficient inference approach, especially with the support of plugins needed for model-specific operations like grid sampling.
-
Gradio App Integration: Users can initiate animations using either Onnxruntime or TensorRT modes, with an easy-to-navigate web application hosted locally.
FasterLivePortrait emerges as a powerful tool for animating portraits in real time. With its cutting-edge features and user-friendly setup, it is poised to become a staple among developers and creatives looking to explore dynamic visualizations and interactive media.