OpenDelta: A Toolkit for Parameter-Efficient Tuning
Overview
OpenDelta is an open-source framework designed to enhance the flexibility and efficiency of machine learning model tuning. Known as delta tuning, this approach allows users to add a small number of parameters for update while keeping most of the model's parameters fixed. This method helps in implementing various tuning techniques such as prefix-tuning, adapters, and Lora, using preferred pre-trained models (PTMs).
Key Features
-
Compatibility: The latest version of OpenDelta has been tested with Python 3.8.13, PyTorch 1.12.1, and transformers 4.22.2, but it may also function with other versions. Users can report bugs for the team to address.
-
Visualization: A demo showing how OpenDelta can modify pre-trained models, such as BART, is provided to illustrate the benefits of delta tuning.
Recent Updates
OpenDelta has undergone several significant enhancements:
- Version 0.3.2 (October 25, 2022): This release includes support for BMTrain and improvements to documentation and inspection utilities.
- Version 0.3.0 (October 14, 2022): This update made the setup of each delta tuning method more user-friendly and ensured compatibility with models that have supported submodules.
- Earlier Fixes (March 2022): Addressed bugs in Soft Prompt Tuning and Prefix Tuning, added a Google Colab example for efficient multitask training, and updated the pip version.
Installation
For those interested in trying OpenDelta, installation can be done via:
-
Creating a Virtual Environment (Optional)
conda create -n opendelta_env python=3.8 conda activate opendelta_env -
Installing the Latest Version
pip install git+https://github.com/thunlp/OpenDelta.git -
Or Installing the Stable Version from pip
pip install opendelta -
Building from Source
git clone [email protected]:thunlp/OpenDelta.git cd OpenDelta python setup.py install
Must-Try Guide
OpenDelta provides comprehensive tutorials through code examples and Jupyter notebooks (both in Python scripts and Google Colab). Users can utilize these resources to understand the key functionalities, such as adding delta models to pre-trained language models like T5, saving and loading delta models, and applying configurations to customized wrapped models.
Supported Models
While OpenDelta is intended for a wide range of PyTorch-based backbone models, it has been explicitly tested on commonly used models to ensure support. This testing is ongoing, and user contributions to extend these capabilities are welcomed.
Conclusion
OpenDelta represents an innovative solution for enhancing the adaptability of pre-trained models. With community involvement and continuous development, it aims to support an increasing variety of models and applications, promoting efficient and effective machine learning practices.