chameleon - Experience High-Performance Mixed-Modal Models with GPU-Optimized Inference

Introduction to Meta Chameleon

Meta Chameleon is an innovative project developed by the FAIR team at Meta AI. It focuses on providing tools and resources for a unique multimodal model that combines various types of data inputs, known as "mixed-modal early-fusion" models.

Key Features and Tools

Inference Code
- Meta Chameleon offers a robust inference implementation designed to run on GPUs. This allows users to quickly execute model checkpoints and evaluate the performance of the AI models.
Input-Output Viewing
- The project includes a viewer tool that can display both the inputs and outputs of the model in a detailed and interactive manner. This tool is accessible via a web browser, enhancing the user's ability to analyze and interpret data efficiently.
Evaluation Prompts
- The repository provides diverse prompts that assist in the assessment of the model's capabilities, offering both mixed-modal and text-only formats for comprehensive human evaluation.

System Requirements

To take full advantage of the Meta Chameleon tools, a computer with a CUDA-capable GPU is necessary. However, for those who prefer other types of hardware, Meta Chameleon has made alternatives compatible with different platforms, available through services like HuggingFace.

Getting Started

Installation of the Meta Chameleon tools is straightforward:

To install the basic components, users can execute a simple command using pip, a package manager for Python.
```
pip install -U git+https://github.com/facebookresearch/chameleon.git
```
To access the full suite of visual tools, cloning the repository and installing from the root directory is required.
```
git clone https://github.com/facebookresearch/chameleon.git
cd chameleon
pip install -e .
```

Before using the models, users must download the required model checkpoints and configurations. Access is granted through a pre-signed URL provided via email upon request.

Utilizing the Viewer

For comprehensive visualizations:

Main Viewer: This tool can be initiated using Docker Compose. It displays detailed input and output interactions.
```
docker-compose up --build
```
By default, it runs a model with 7 billion parameters, though users can modify the configuration to explore other models.
MiniViewer: A smaller, lightweight version of the viewer is also available. It offers similar functionalities for debugging purposes.
```
python -m chameleon.miniviewer
```
Users can switch to a larger model by specifying the model size in the command.

License and Citation

The Meta Chameleon project and its resources are shared under the Chameleon Research License, with guidelines detailed in the accompanying LICENSE file. For those wishing to cite the Meta Chameleon work in academic papers or research, a specific citation format is provided.

Meta Chameleon provides a cutting-edge platform for working with early-fusion models, allowing users to explore complex multimodal data efficiently. With its user-friendly setup and visualization tools, it is a valuable resource for researchers and professionals interested in advanced AI model capabilities.