awesome-rust-llm - Explore Leading Rust Libraries and Tools for Large Language Models

Introduction to Awesome Rust LLM

"Awesome Rust LLM" is a comprehensive and curated list dedicated to Rust-based frameworks, libraries, tools, tutorials, and articles that are designed for Machine Learning models. It acts as a hub for developers to discover the best resources and contribute their own work through pull requests.

Models & Inference

llm: This Rust library is essential for running model inferences. It supports various Language Model (LLM) models and manages models based on ggml.
rust-bert: Offering all-in-one pipelines for transformer-based models like BERT, DistilBERT, and GPT2, this library is perfect for tasks like local embedding and serves as a port of Python's "transformers".
llm-chain: It allows for the chaining of LLMs within Rust, providing a flexible LLM management framework.
smartgpt: This tool gives LLMs the ability to perform complex tasks via plugins, enhancing their capability beyond standard usage.
diffusers: An impressive solution for using Stable Diffusion in Rust, operating 45% faster than its PyTorch equivalent.
postgresml: A Postgres extension that integrates model fetching and inference directly within your SQL commands in a Postgres instance.

Projects

Here are notable projects built using Rust in the LLM space:

aichat: A Rust-based command-line interface (CLI) for AI chat, featuring real-time streaming, text highlighting, and more.
browser-agent: Uses GPT-4 to interact with a headless browser by sending and executing simplified page instructions.
tenere: Provides a text-based user interface (TUI) for managing LLMs efficiently.
ajeto: Functions as a personal assistant powered by LLMs, offering a layer of personalized, smart interactions.
shortgpt: Answers your questions instantly and concisely.
autorust: Employs macros to generate AI-driven Rust code at compile time.
clerk: A sophisticated file organizer driven by LLM intelligence.
gptcommit: Assists in creating precise and useful commit messages using GPT.

LLM Memory

indexify: A long-term memory and retrieval service for LLMs, ensuring efficient data management.
memex: This tool offers a Rust-powered memory management system focusing on semantic search and document storage.
motorhead: Acts as both a memory and information retrieval server for LLMs. It uses Redis for vector storage and is compatible with the OpenAI API.

Core Libraries

tiktoken: A tokenization library from OpenAI with a Rust core to handle BPE, tailored for efficient performance with OpenAI models.

use tiktoken_rs::p50k_base;

let bpe = p50k_base().unwrap();
let tokens = bpe.encode_with_special_tokens(
  "This is a sentence with spaces"
);
println!("Token count: {}", tokens.len());

tiktoken-rs: Expands on tiktoken, offering Rust-specific enhancements and functionalities.
polars: A high-performance, pure Rust alternative to pandas for data manipulation tasks.
rllama: Implements LLaMa inference in pure Rust, useful for embedding into applications or wrapping in another programming language.
whatlang: A language detection library using logistic regression models, making language identification seamless.
OpenAI API: A robust client for interfacing with the OpenAI API from Rust, ensuring type safety and ease of use.

Tools

spider: A Rust-based web crawler, ideal for creating comprehensive website dumps, and initiating post-processing at a later stage.

Services

dust: Offers a full service for workflow automation using composable blocks. Its core is developed in Rust with various frontends designed in Typescript.

Vector Stores

pgvecto.rs: A vector database plugin for Postgres, designed for LLMs, which significantly outperforms pgvector in speed by a factor of 20.
qdrant: A forward-thinking vector database tailored for the next wave of AI applications.

This array of libraries, tools, and projects presents a robust ecosystem for Rust in the machine learning landscape, offering both performance and versatility for developers looking to harness the power of language models.