IDM-VTON: Enhancing Diffusion Models for Real-World Virtual Try-on
IDM-VTON is a groundbreaking project aimed at improving diffusion models for virtual try-on applications. This project targets the complex task of creating authentic virtual try-on experiences that work seamlessly in diverse, real-world scenarios.
Overview
The IDM-VTON project is the official implementation of the research paper titled "Improving Diffusion Models for Authentic Virtual Try-on in the Wild." It focuses on refining diffusion models to offer more realistic and practical solutions for virtual fashion try-ons. The team has provided demo models, along with inference and training codes, to help developers and researchers experiment and build upon their work.
Data Preparation
To train the models used in IDM-VTON, specific datasets like VITON-HD and DressCode are employed:
- VITON-HD: A dataset structured to include images, densepose images, agnostic masks, and clothing items, which are divided into training and testing directories.
- DressCode: Another dataset featuring images and densepose mappings, with related caption files for garment descriptions.
These datasets are crucial for teaching the models to understand and interpret clothing on various body types and poses.
Training Process
The training process requires preparation with pre-trained models and adapters. IDM-VTON utilizes the IP-Adapter, a sophisticated tool tailor-fitted for efficient diffusion model training. The training script can be run directly with specific parameters to fine-tune the models on the datasets.
Inference and Implementation
For inference, IDM-VTON provides specific scripts tailored for different datasets:
- VITON-HD Inference: This uses a detailed command to generate virtual try-on results from the test data.
- DressCode Inference: You can specify categories such as upper body to direct the model on which clothing type to focus during generation.
The project supports a local demonstration through Gradio, a tool allowing users to interact with the model in real-time. This setup requires additional checkpoints for human parsing to ensure the model's accuracy in rendering human figures and clothing accurately.
Acknowledgements and Contributions
The IDM-VTON project is supported by various platforms and codes, including ZeroGPU for providing free GPU resources. It also builds upon several base codes and inspirations like IP-Adapter, OOTDiffusion, DCI-VTON for masking generation, SCHP for human parsing, and Densepose for human body mapping.
Licensing
The codes provided by IDM-VTON are under the CC BY-NC-SA 4.0 license, making them available for non-commercial use, with appropriate credits and condition sharing.
IDM-VTON represents a leap forward in making virtual try-on technology more practical and realistic. It opens up new possibilities in fashion technology by offering developers robust tools and datasets to innovate and improve how virtual clothing is depicted on human models.