First Order Motion Model for Image Animation
The "First Order Motion Model for Image Animation" is an open-source project developed by researchers Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. This project stands out in the field of image animation, allowing the transformation of static images into animated sequences by transferring the motion from a driving video to the image subject.
Example Animations
The project showcases its capabilities using various datasets. For instance, the VoxCeleb dataset demonstrates animations where the motion from a driving video is applied to a static image of a person, making the still subject appear dynamic. Similar examples are available for fashion and MGIF datasets, illustrating the versatility of the model.
Installation and Setup
The project supports Python 3 and requires several dependencies. These can be installed by running:
pip install -r requirements.txt
Configuration for different datasets is managed via YAML files, such as taichi-256.yaml, where parameters for each dataset are defined.
Pre-trained Checkpoints and Running a Demo
Pre-trained models (or checkpoints) are provided, which can be downloaded and used to quickly test the animation capabilities. To run a demo with a pre-trained checkpoint, the following command can be used:
python demo.py --config config/dataset_name.yaml --driving_video path/to/driving --source_image path/to/source --checkpoint path/to/checkpoint --relative --adapt_scale
The output will be saved as result.mp4.
For those encountering compatibility issues, the demo can also be run using Docker, ensuring a controlled environment.
Online Demos
For an easier start, there are public demos on platforms like Google Colab and Kaggle, which allow users to experiment with the model without local setup. The Colab demo even offers a graphical user interface.
Animation Modes
The project supports different animation techniques using either absolute or relative keypoint coordinates. The relative approach tends to deliver better results by maintaining the natural pose of the subject from the source image.
Training and Evaluation
The repository provides detailed instructions for training the model on new datasets, evaluating its performance, and using it for tasks such as face-swapping. Training involves setting up the data, defining configurations, and running the training scripts.
Supported Datasets
Several datasets are supported, such as Bair, MGIF, Fashion, Taichi, Nemo, and VoxCeleb, each requiring specific preprocessing steps. The dataset preparation instructions are comprehensive, facilitating smooth dataset integration for training.
Custom Dataset Training
Users aiming to train on custom datasets should resize videos to a consistent format and organize them into training and testing folders. A custom configuration file should also be set up to match the dataset specifics.
Citation
For academic purposes, users of this project are encouraged to cite the original paper:
@InProceedings{Siarohin_2019_NeurIPS,
author={Siarohin, Aliaksandr and Lathuilière, Stéphane and Tulyakov, Sergey and Ricci, Elisa and Sebe, Nicu},
title={First Order Motion Model for Image Animation},
booktitle = {Conference on Neural Information Processing Systems (NeurIPS)},
month = {December},
year = {2019}
}
This project represents an innovative method for image animation, leveraging motion transfer to breathe life into still images across multiple domains and applications.