Knowledge-Distillation-Toolkit
The Knowledge Distillation Toolkit is a solution for compressing machine learning models with knowledge distillation, tailored for use with PyTorch and PyTorch Lightning. The toolkit supports the implementation of teacher and student models, data loaders for both training and validation processes, and an inference pipeline for performance evaluation. Designed to minimize model size while maintaining accuracy, it enables efficient knowledge transfer from a larger, complex model to a smaller student model. The toolkit also offers flexible configuration options such as customizable architectures, optimization methods, and learning rate scheduling to refine the model compression workflow.
