catai - Efficient AI Model Management on Your Computer

Introduction to CatAI

CatAI is a versatile AI platform that allows users to run GGUF models right on their computer, providing them with a personal AI assistant available offline. It is inspired by projects like Node-Llama-Cpp and Llama.cpp, aiming to make AI assistance more accessible and efficient.

Installation & Usage

To use CatAI, users need to have Node.js installed on their computers. The installation process involves a simple command-line operation:

npm install -g catai

catai install meta-llama-3-8b-q4_k_m
catai up

This installs the necessary components and sets up a chat interface where users can interact with their AI assistant.

Features

CatAI comes with several user-friendly features, such as:

Auto-detection of Programming Language: It can automatically detect the programming language being used, making it easier to integrate AI capabilities.
Original Message Display: Users can click on a user icon to view the original message.
Real-time Text Streaming: Interactions happen in real time, providing immediate responses.
Fast Model Downloads: The platform supports rapid downloading of models, enhancing user efficiency.

Command-line Interface (CLI)

CatAI provides a robust CLI with several functionalities. Users can install models, list available models, set active models, start the chat service, update the server, and manage the installed models. For instance, the serve|up command opens the chat website, allowing interaction with the AI.

Cross-Platform Compatibility

CatAI is designed to work across various operating systems, including Windows, Linux, and Mac. It leverages the node-llama-cpp package, supporting a range of platforms like darwin-x64, linux-x64, and win32-x64-msvc.

Web API

The platform offers a simple web API for developers to query models and receive responses programmatically. This API can be accessed through a basic HTTP request, allowing for flexibility in usage.

const response = await fetch('http://127.0.0.1:3000/api/chat/prompt', {
    method: 'POST',
    body: JSON.stringify({ prompt: 'Write me 100 words story' }),
    headers: { 'Content-Type': 'application/json' }
});

const data = await response.text();

Development API

For developers interested in a more in-depth interaction, CatAI offers a development API. This enables the integration of AI models into custom applications and environments using commands and predefined schemas.

Configuration

Users can customize their CatAI experience through a web interface, adjusting settings to fit their specific needs. More detailed information on configuration options is available in the documentation.

Contribution and Licensing

CatAI encourages community contributions. Developers interested in contributing can refer to the project's contributing guide. It's important to note that CatAI utilizes Llama.cpp for running models, and any licensing terms applicable to Llama.cpp also apply to CatAI.

In summary, CatAI provides a powerful and flexible platform for running AI models on personal computers, offering users and developers a wide array of tools and interfaces for customization and integration.