Project Introduction to "Talking Head Anime from a Single Image"
The "Talking Head Anime from a Single Image" project offers a fascinating glimpse into the capabilities of neural networks to animate an anime character from a static image. This project, created by Pramook Khungurn, provides two primary tools packaged within a demonstration code repository. For those interested in newer developments, a more advanced Version 2 of this software can be found at: Talking Head Anime from a Single Image Version 2.
Applications of the Project
- Manual Poser:
- This tool allows users to manually control the pose of an anime character. By adjusting sliders, users can manipulate different aspects of the character's appearance and posture.
- Puppeteer:
- In this application, users can animate an anime character by having it mimic human head movements captured via a webcam. This real-time integration brings the character to life as it mirrors the actions of a human subject.
Trying the Manual Poser
For those who wish to experiment with the Manual Poser without investing in specific hardware or setting up a local environment, the application is hosted on Google Colab. This accessible platform allows users to run the application online without needing a powerful PC. Simply visit the Google Colab link to get started.
Hardware and Software Requirements
Running this project requires a robust and modern Nvidia GPU primarily to handle the computational requirements effectively. Successful instances have used NVIDIA Geforce GTX 1080 Ti and Titan RTX GPUs. Additionally, a webcam is necessary for utilizing the puppeteer tool.
The project's code is built using Python and necessitates several key dependencies, including:
- Python 3.6 or higher
- PyTorch 1.4.0 or higher
- Dlib 19.19 or higher
- OpenCV-Python 4.1.0.30 or higher
- Pillow 7.0.0 or higher
- Numpy 1.17.2 or higher
These libraries form the backbone of the project's animation and face-tracking capabilities.
Setting Up with Anaconda
Users of the Anaconda distribution of Python can conveniently recreate the required environment by utilizing an existing environment configuration file provided in the project. This setup provides a standardized workspace with all necessary modules installed, although it does come with a requirement of substantial storage space.
To create the working environment, users can execute:
conda env create -f environment.yml
Once installed, activate the environment using the command:
conda activate talking-head-anime
Deactivate when finished with:
conda deactivate
Preparing the Data
Essential to this project's operation is the proper setup of model data. After cloning the repository, one must download the necessary model files. The key files include:
- Various .pt files needed for animation processing
- Shape predictor for facial landmark detection
Character images to be animated must be in a specific format: PNG, 256 x 256 pixels, with a centered character head and a transparent background. Users can generate and edit their anime characters to meet these criteria, with Waifu Labs as a recommended tool for creating suitable images.
Running the Program
Once your setup is complete:
- To start the Manual Poser, run
python app/manual_poser.py - To activate the Puppeteer tool, use
python app/puppeteer.py
Academic Use and Citation
For academic references or projects that benefit from this code, please cite the work accordingly using the official project website: Talking Head Anime from a Single Image.
Citation format:
- BibTex and other citation styles are provided in the project documentation for academic use.
Note on Ownership and Usage
This software, while developed by Pramook Khungurn during his tenure at Google Japan, is not affiliated with Google's products and comes with personal liability for its release and use.