#Mixture of Experts

Mergoo is a versatile library that facilitates the merging and training of multiple LLM experts. It features methods like Mixture-of-Experts and Layer-wise merging, providing layer-specific flexibility. Compatible with models such as Llama, Mistral, Phi3, and BERT, Mergoo supports various platforms including CPU, MPS, and GPU. It offers tools for training the router of MoE layers or fully fine-tuning integrated LLMs. Discover integration methods through tutorials and engage with the open-source community for further contributions.

DiT-MoE offers a scalable and efficient solution with its PyTorch implementation of Sparse Diffusion Transformers, designed to handle up to 16 billion parameters. Featuring advanced techniques like rectified flow-based training and expert routing, it aids in reducing computational load while enhancing model accuracy and convergence, with practical support from DeepSpeed. This project provides valuable assets such as pre-trained models and detailed scripts, catering to researchers requiring flexible and high-performing AI frameworks.

This repository offers JAX example code to run the Grok-1 model, characterized by its Mixture of 8 Experts architecture and 314 billion parameters. The process requires significant GPU memory and involves 64 layers and 48 attention heads. The SentencePiece tokenizer encompasses 131,072 tokens with features like rotary embeddings, activation sharding, and 8-bit quantization for sequences up to 8,192 tokens. Weights can be downloaded through a magnet link or the HuggingFace Hub under the Apache 2.0 license.

Discover the Pytorch implementation of Sparsely Gated Mixture of Experts intended to enhance language model capacity by increasing parameters without additional computation. This version adds features to the original TensorFlow model, supporting complex architectures such as hierarchical mixtures, and enables customization of expert networks with various activation functions and gating policies. Suitable for developers who wish to scale models effectively while maintaining performance, it includes setup and usage instructions for easy integration.