SupContrast: Supervised Contrastive Learning
SupContrast is a project focused on implementing supervised contrastive learning methods primarily using the PyTorch framework. This allows developers and researchers to leverage the power of contrastive learning to improve models, especially in visual representation tasks. The repository provides examples using CIFAR datasets to illustrate these methods effectively.
Background
The SupContrast project references two influential papers:
- Supervised Contrastive Learning - This paper introduces a framework for enhancing learning paradigms through contrastive loss while utilizing labelled data.
- A Simple Framework for Contrastive Learning of Visual Representations - A foundational paper that propounds the advantages and methodology of contrastive learning for visual tasks.
Key Updates
The project now includes a simplified implementation of the supervised contrastive loss, known as SupCon Loss. Interestingly, this loss function is akin to a cross-entropy loss, which should provide ease of understanding and implementation for users acquainted with traditional loss functions. For clarity and simplicity, a cleaner implementation is available via Google Research.
Additionally, the project has released an ImageNet model that demonstrates significant performance, achieving over 79% top-1 accuracy using a small batch size in combination with a momentum encoder.
Loss Function
The SupConLoss function is central to this project, taking L2 normalized features and labels as inputs. If labels are omitted, it reverts to SimCLR, catering to unsupervised scenarios. This flexibility allows for broader application and ease in experimenting with different datasets and scenarios.
Performance and Comparisons
The effectiveness of SupContrast is evident across various datasets such as CIFAR-10, CIFAR-100, and ImageNet. Here's a snapshot of its performance:
- CIFAR-10: Achieving an impressive 96% accuracy with ResNet50 architecture when using the supervised contrastive learning paradigm.
- CIFAR-100: Displays a 76.5% accuracy, again outshining traditional supervised cross-entropy methods.
- ImageNet: Boasts a noteworthy 79.1% accuracy with the innovative use of the MoCo trick.
Running the Models
To utilize SupContrast, users might need to configure GPU settings using CUDA_VISIBLE_DEVICES and decide between CIFAR-10 or CIFAR-100 datasets. The process involves different stages for model execution, from standard cross-entropy to pretraining with supervised contrastive learning, as well as using SimCLR for unsupervised experiments.
Visualization with t-SNE
Visualizations using t-SNE offer insights into how models trained with SupContrast, Standard Cross-Entropy, and SimCLR differ. These visualizations can be instructional for both novice and experienced researchers to comprehend the discriminative power of different learning methods.
Conclusion
SupContrast is a robust project that not only provides high-performance models for visual tasks but also simplifies the implementation of complex learning paradigms. By focusing on supervised contrastive learning, it bridges the gap between unsupervised learning benefits and the utilization of labeled data, proving to be an invaluable tool for advancing machine learning research.