Awesome-Parameter-Efficient-Transfer-Learning - Explore Resources on Efficient Transfer Learning Techniques for Pre-Trained Models

Awesome Parameter-Efficient Transfer Learning

Introduction

The Awesome Parameter-Efficient Transfer Learning project is a comprehensive collection of resources dedicated to the field of parameter-efficient transfer learning. This area of machine learning focuses on optimizing the transfer of knowledge from a pre-trained model to a new task using as few parameters as possible. Such efficiency is crucial for deploying large models in environments with limited computational resources, like mobile devices or embedded systems.

Project Highlights

The repository showcases a variety of techniques aiming to make fine-tuning of pre-trained vision models more efficient. It is particularly useful for researchers and developers working on applications that require adapting large models to specific tasks without requiring extensive computational power or wasting resources.

Important Updates

March 2024: The Visual PEFT Library/Benchmark repository was launched, providing tools and benchmarks for visual models.
February 2024: The team released a survey titled Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models, which provides valuable insights into the current state and advancements of the field.
January 2023: The Awesome-Parameter-Efficient-Transfer-Learning repository was established, marking the beginning of this resourceful project.

Citation

For researchers wishing to cite this repository or the survey in their work, below is the recommended citation format:

@article{xin2024parameter,
  title={Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey},
  author={Xin, Yi and others},
  journal={arXiv preprint arXiv:2402.02242},
  year={2024}
}

Content Overview

Keywords: This section provides a list of key terms and concepts relevant to the field of parameter-efficient transfer learning.
Papers: A collection of research papers organized under various tuning methods, such as:
- Addition-based Tuning
  - Adapter Tuning: A method with around 30 papers highlighted.
  - Prompt Tuning: Another focus area with relevant research papers.

The project aims to serve as a valuable reference for the academic community, offering up-to-date information and resources on efficient transfer learning techniques. Its structured approach and comprehensive categorization of papers and methods make it a go-to repository for those interested in optimizing machine learning models.