Online Test-time Adaptation: Project Introduction
Overview
The online test-time adaptation project is an open-source repository developed by Robert A. Marsden and Mario Döbler, utilizing the PyTorch framework. This repository serves as the main hub for several research papers focusing on the novel techniques of adapting machine learning models during test-time. These works include strategies like self-training, weight ensembling, and style transfers, ensuring model robustness and effectiveness even after deploying them.
Key Research Contributions
This repository is central to several groundbreaking studies:
- Self-Training with Intermediate Domains: This study introduces intermediate domains to improve self-training processes during test-time.
- Robust Mean Teacher: A technique focusing on a gradual and continual adaptation strategy, presented at the CVPR2023.
- Universal Test-time Adaptation: An approach through ensemble methods and correction techniques, scheduled for presentation at WACV2024.
- Vision-Language Model Adaptation: A comparative study exploring opportunities in vision-language learning models, prepared for CVPR2024.
Prerequisites
To utilize this repository, users are provided a convenient conda environment setup. The steps include updating conda, creating an environment from the provided YAML file, and finally activating this environment for use.
Features
Datasets
The repository encompasses a wide variety of datasets, including:
- CIFAR10-C, CIFAR100-C
- Various ImageNet derivations such as ImageNet-C, ImageNet-A, ImageNet-R, and others
- DomainNet-126 and Continually Changing Corruptions (CCC)
Models
The repository supports various pre-trained models from platforms like Torchvision and Timm. These models cater to major datasets, particularly ImageNet variations and DomainNet-126. Some models include ResNet-26 GN and OpenCLIP models.
Settings
A range of experimental setups are available, such as:
- Reset the model after each domain shift
- Continual and gradual domain adaptation
- Handling mixed domains and correlated samples
Methods
A broad array of methods is integrated, touching on state-of-the-art approaches like TENT, MEMO, ETA, CoTTA, and others. These methods support CLIP models, and the repository promotes a modular design facilitating easy addition of new methods.
Mixed Precision Training
Many methods in the repository benefit from mixed precision training, which accelerates experiments and reduces memory usage. However, benchmark results are usually generated using fp32 precision.
Getting Started
Running Benchmarks
To conduct experiments, the relevant datasets must be downloaded. Different benchmarks cater to particular datasets and settings, with explicit guidance for setting up and modifying configurations as necessary in the repository's configuration files.
Configurations
The evaluation configuration can be easily modified with specific commands to select different models, datasets, or experimental setups. This ease of configuration allows for versatile experimentation.
Benchmarking and Results
A comprehensive benchmarking suite provides detailed results for each method across different models and settings. This suite is continually updated as new methodologies or datasets are introduced. Additionally, further insights are documented in the linked research paper.
Contributions and Acknowledgements
The project welcomes contributions, encouraging pull requests to integrate new methods. It acknowledges related works and sources from other projects like Robustbench, CoTTA, TENT, and others. This collaborative openness lays the foundation for continual advancements in test-time adaptation techniques.
For those interested in segmentation, the repository includes extensions based on CarlaTTA datasets and related models, demonstrating versatility across different tasks.
Overall, this project serves as a significant resource for those exploring and enhancing model performance at test-time, reinforcing the adaptability and resilience of machine learning systems in varied and dynamic data environments.