#speculative decoding

Medusa provides a framework for enhancing the efficiency of LLMs through the use of multiple decoding heads, achieving up to 3.6x acceleration. It maintains the original model unchanged and employs a tree-based attention mechanism to improve non-greedy generation, overcoming speculative decoding inefficiencies. This system supports parameter-efficient training, making it accessible for limited GPU setups. Recent updates include Medusa-2 for full-model training and self-distillation features that enable integration with various fine-tuned LLMs without needing original training data.

Explore ExUI, a lightweight browser-based UI for running local inference with the ExLlamaV2 framework. It offers a minimalistic, responsive interface with persistent sessions, multiple instruction formats, and supports EXL2, GPTQ, and FP16 models. The platform includes a notepad mode and is compatible with Google Colab. Installation is easy with prebuilt wheels and latest Flash Attention recommendations.