#AI Inference

Discover a minimalist C/C++ system for Stable Diffusion and Flux inference, seamlessly integrating with tools like ggml and supporting a wide range of versions including SD1.x, SD2.x, and SDXL. Inspired by llama.cpp, the project enhances memory efficiency and accelerates CPU and GPU performance via CUDA, Metal, Vulkan, and SYCL. It offers comprehensive support for diverse weights, easy quantization, and intuitive sampling methods, presenting a versatile and optimized solution for developers. With compatibility across Linux, Mac OS, Windows, and Android, this project ensures broad accessibility and integration options.

This open-source AI framework offers a native alternative to OpenAI's API, featuring text generation and speech-to-text capabilities across Windows, MacOS, and Linux. Designed for on-device AI tasks, it prioritizes data privacy and eliminates the need for cloud infrastructure. With model caching, session management, and GPU support to boost performance, it serves both individual projects and scalable enterprise solutions, emphasizing user autonomy and private data handling.

YoMo is an open-source framework for building geo-distributed AI applications, emphasizing low latency and strong security. It leverages the QUIC transport protocol and a stateful serverless architecture to provide efficient and reliable AI inference closer to users. Highlights include low-latency guaranteed by QUIC, TLS v1.3 security for all data packets, and a geo-distributed architecture that enhances AI inference proximity. The framework includes support for the Y3 real-time codec to boost performance. YoMo is well-suited for fast and secure AI deployment, offering extensive documentation and community support, catering specifically to developers working on AI infrastructure.