FollowYourPose: Pose-Guided Text-to-Video Generation
The FollowYourPose project is an innovative step in digital video generation, aiming to create videos guided by text and pose inputs. This groundbreaking research, published in AAAI 2024, addresses the challenge of generating videos of characters that can be controlled by specific poses described in text.
Concept and Innovation
The core idea of FollowYourPose is to use non-paired video data and pre-trained text-to-image models to generate videos that can be manipulated using text and pose descriptions. Traditionally, creating such videos required datasets with detailed pose annotations paired with videos, which are hard to come by. FollowYourPose bypasses this limitation by using a two-stage training approach.
-
Stage One: Controlled Text-to-Image Generation
- This stage involves using pairs of images and keypoint data to enable text-to-image generation that is pose-controllable. A specialized encoder is trained to capture pose information, allowing for a zero-initialized convolutional process that integrates pose data into the video generation model.
-
Stage Two: Enhancing Motion with Pose-Free Videos
- In the second stage, the system fine-tunes motion using unannotated videos. It achieves this by incorporating temporal and cross-frame self-attention mechanisms, allowing it to maintain the narrative control and integration capabilities of its text-to-image roots.
Features and Demonstrations
The project demonstrates its capabilities through a variety of examples. It features characters ranging from astronauts to fictional superheroes being animated within various environments, all generated from straightforward text descriptions and pose inputs. These demonstrations are made accessible through platforms like Hugging Face and Google Colab, allowing enthusiasts and developers to experiment with the technology.
Technical Infrastructure
FollowYourPose is built to run efficiently on NVIDIA A100 or 3090 GPUs, utilizing CUDA and xformers for acceleration. For developers interested in experimenting with the project, the setup involves basic Python environment preparation and installation of required components via requirements.txt. Additionally, users can explore the demo locally using Gradio, provided they have the necessary hardware.
Results and Community Engagement
The results produced by FollowYourPose exhibit a blend of realistic environments and imaginative concepts, crafted through the lens of user's poses and textual descriptions. The community is encouraged to explore these outputs, share feedback, and contribute by starring the project's GitHub repository.
Future and Extensions
While the current iteration of FollowYourPose showcases remarkable results, the team envisions expanding its applications. This includes enhancing the technology to incorporate more intricate motions and scenarios, making digital human creation and animation more accessible and versatile.
In summary, FollowYourPose represents a significant advancement in text-to-video generation, offering new possibilities for creators and researchers in the field of digital media and animation.