LLaMA2-Accessory: Catalyzing LLM Development
LLaMA2-Accessory is a public and free toolkit aimed at assisting in the development of Large Language Models (LLMs) and multimodal LLMs. This toolkit has been enhanced from its predecessor, LLaMA-Adapter, providing a suite of advanced features for enthusiasts and developers.
Introducing SPHINX
One of the significant contributions of LLaMA2-Accessory is the introduction of SPHINX, a sophisticated multimodal LLM. It integrates a wide range of training tasks, covers various data domains, and supports diverse visual embeddings, making it highly versatile for different applications.
Latest News and Updates
- March 7, 2024: The toolkit unveiled demos and the codebase for Large-DiT-T2I.
- February 17, 2024: Released 3 and 7 Billion Large-DiT models trained on ImageNet, along with pre-trained checkpoints and the full training codebase.
- January 27, 2024: SPHINX-MoE achieved notable accuracy improvements on benchmarks such as CMMMU-test and CMMMU-val.
- January 24, 2024: Delivered a new state-of-the-art performance on MMVP, surpassing even GPT-4V.
- New Models and Features: Introduced models like SPHINX-Tiny and started supporting frameworks such as OpenCompass and mixtral-8x7b.
Key Features
- Expanded Dataset and Task Support: The toolkit accommodates a range of datasets for pretraining and finetuning, supporting tasks from single-modal to multi-modal finetuning.
- Efficient Optimization and Deployment: Implements efficient finetuning strategies and supports advanced methods like Fully Sharded Data Parallel (FSDP) and QLoRA.
- Support for Various Visual Encoders and LLMs: Integrates visual encoders such as CLIP and LLMs like LLaMA2, CodeLlama, and Falcon, to meet diverse research and development needs.
Setup and Usage
Instructions for setting up the environment, model pretraining, finetuning, and inference are thoroughly documented, ensuring users can quickly get started with their projects.
Demos and Applications
The toolkit offers several demos to showcase its capabilities, including instruction-tuned and chatbot versions of LLaMA2, and the multimodal capabilities of models like SPHINX, which can generate high-quality image annotations.
Core Team and Acknowledgements
The project is driven by contributors from the General Vision Group at the Shanghai AI Lab, with leaders Peng Gao, Wenqi Shao, and Shanghang Zhang at the helm. The success of LLaMA2-Accessory is underpinned by contributions from various open-source projects and organizations, enhancing its functionality and outreach.
Opportunities and Further Information
For those interested in joining the continued development of LLaMA2-Accessory, the General Vision Group is hiring positions focused on multi-modality and vision foundation models.
In conclusion, LLaMA2-Accessory provides a comprehensive, community-driven toolkit for advancing the field of language models, inviting exploration, adaptation, and innovation from researchers and developers worldwide.