LLamaSharp

Introduction to LLamaSharp

LLamaSharp is an innovative cross-platform library that enables users to efficiently run LLaMA/LLaVA and other language models on local devices. Drawing inspiration from llama.cpp, LLamaSharp provides robust support for both CPU and GPU operations, offering an enhanced experience for deploying large language models (LLMs) seamlessly into various applications. This library is designed with high-level APIs and Retrieval Augmented Generation (RAG) support to make integration into your software both simple and effective.

Features and Capabilities

• Cross-Platform Compatibility: LLamaSharp is versatile and functions smoothly on different operating systems, including Windows, Linux, and Mac.

• Efficient Inference: With capabilities on both CPU and GPU, LLamaSharp ensures your language models perform optimally, no matter the device.

• High-Level APIs: These simplified APIs accelerate the process of deploying LLMs in your applications, saving time and effort.

• Support for RAG: LLamaSharp supports Retrieval Augmented Generation, enabling more advanced and accurate AI responses by enhancing data retrieval processes.

Installation and Setup

Setting up LLamaSharp involves interacting with native backend libraries for optimal performance. These libraries are available for different platforms and hardware capabilities, such as pure CPU, CUDA for GPU acceleration, and Vulkan. Users can install these backend packages directly without the need for compiling C++ code themselves.

To get started:

1. Install LLamaSharp:

   PM> Install-Package LLamaSharp
   

2. Select and Install Backend: Choose the appropriate backend for your system:
   • LLamaSharp.Backend.Cpu (CPU support, including Metal for Mac)
   • LLamaSharp.Backend.Cuda11 or Cuda12 (for CUDA support on Windows & Linux)
   • LLamaSharp.Backend.Vulkan (Vulkan support for Windows & Linux)

For further integrations, consider using LLamaSharp.semantic-kernel for seamless Microsoft semantic-kernel integration, or LLamaSharp.kernel-memory for RAG capabilities, which requires .NET 6.0 or higher.

Model Preparation

LLamaSharp employs a specific GGUF model file format, which can be generated from widely-used formats like PyTorch or Huggingface. To obtain a GGUF file:

1. Download Pre-converted Models: Search for models labeled 'gguf' on platforms like Huggingface.
2. Convert Models Manually: Use python scripts as detailed in the llama.cpp readme to convert models from PyTorch/Huggingface formats.

Example: Starting a Chatbot

With LLamaSharp, developers can create chatbots that interact naturally with users. Here's a brief example of initializing a chat session with a model, demonstrating the ease of use LLamaSharp offers.

using LLama.Common;
using LLama;

// Define model path and parameters
string modelPath = @"<Your Model Path>";
var parameters = new ModelParams(modelPath) { ContextSize = 1024, GpuLayerCount = 5 };

// Load Model and Create Session
using var model = LLamaWeights.LoadFromFile(parameters);
using var context = model.CreateContext(parameters);
var executor = new InteractiveExecutor(context);

// Maintain Chat History
var chatHistory = new ChatHistory();
chatHistory.AddMessage(AuthorRole.System, "Describe your chat behavior.");

// Establish Session with Inference Parameters
ChatSession session = new(executor, chatHistory);
InferenceParams inferenceParams = new InferenceParams() { MaxTokens = 256 };

Console.ForegroundColor = ConsoleColor.Yellow;
Console.WriteLine("User: ");
string userInput = Console.ReadLine() ?? "";

// Interactive Loop
while (userInput != "exit") {
    await foreach (var text in session.ChatAsync(new ChatHistory.Message(AuthorRole.User, userInput), inferenceParams)) {
        Console.Write(text);
    }
    userInput = Console.ReadLine() ?? "";
}

Community and Contribution

LLamaSharp is open for contributions and boasts a vibrant community. Contributions are warmly welcomed whether through feedback, feature requests, or direct involvement in development. Users are encouraged to join the conversation on platforms such as Discord and engage with like-minded developers in the active LLamaSharp community.

For more examples, extensive FAQs, and our detailed documentation, visit our official documentation. Whether you're a novice or a seasoned developer, LLamaSharp provides the tools and support to integrate cutting-edge natural language processing into your projects efficiently.