Introduction to FollowYourEmoji
Overview
The FollowYourEmoji project is an innovative framework designed to bring portraits to life through animation. This project is remarkable for its ability to animate a static reference portrait using a sequence of landmarks. Essentially, this means users can create animated versions of portraits by defining the movement and expressions through recognizable patterns called landmarks.
Key Features
- Diffusion-Based Framework: FollowYourEmoji relies on a diffusion-based process, making the animations smooth and natural.
- Fine-Controllable Animation: Users have considerable control over the animation process, allowing for precise adjustments and customizations to create exactly the expressions and movements desired.
- Expressive Freestyle Portrait Animation: The framework supports a broad range of facial expressions, enabling users to create highly expressive portraits.
Project Release and Recognition
This cutting-edge framework was accepted as a significant contribution to Siggraph Asia 2024, highlighting its importance and relevance in the field of computer graphics and animation.
Getting Started with FollowYourEmoji
Step 1: Environment Setup
To begin using FollowYourEmoji, users must first clone the code repository and set up the environment. This can easily be done using the following command:
pip install -r requirements.txt
Step 2: Download Pretrained Models
For optimal performance, users should download the pretrained models from Huggingface and organize them in the specified directory structure. A range of models, including base model, vae, and AnimateDiff, are needed to fully utilize the tool.
Step 3: Running Inference
Once the environment is prepared and models are downloaded, the framework is ready to animate portraits. Users run inference using simple bash commands to convert their images and landmark sequences into animated portraits.
Creating Custom Emojis
FollowYourEmoji allows users to create personalized emoji animations. This involves preparing an emoji template using MediaPipe, followed by running the inference with their own video input to generate animations in .npy format.
Follow Family
The creators behind FollowYourEmoji have developed other related projects aimed at animation and video editing:
- Follow-Your-Pose: A tool for pose-guided text-to-video generation.
- Follow-Your-Click: Enables regional image animation using short prompts.
- Follow-Your-Handle: Focuses on controllable video editing through handle transformation.
Citation
If the project is beneficial to research efforts, it is encouraged to cite the work using the provided BibTeX format:
@article{ma2024follow,
title={Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation},
author={Ma, Yue and others},
journal={arXiv preprint arXiv:2406.01900},
year={2024}
}
Summary
In summary, FollowYourEmoji is an advanced tool that revolutionizes how we animate static portraits, transforming them into dynamic, expressive art. With its comprehensive control features and ease of use, it sets a new standard in portrait animation technology.