kani - Enhanced Framework for Flexible Language Model Customization

Introduction to the Kani Project

Overview

Kani, symbolized as "カニ", is a lightweight and adaptable framework tailored specifically for chat-based language models that can utilize tools and execute function calls. Its primary design goal is to offer users a flexible and customizable system that stands out from other language model frameworks, which are often rigid in their configurations. This makes Kani particularly suitable for researchers in natural language processing (NLP), as well as developers and hobbyists.

Supported Models

Kani supports various language models and provides a platform to easily integrate more. Here's a rundown of the models it supports:

Hosted Models

OpenAI Models: This includes versions such as GPT-3.5-turbo, GPT-4, and its various iterations.
Anthropic Models: Models like Claude and Claude Instant fall into this category.

Open Source Models Kani embraces all chat models available on Hugging Face, utilizing libraries like transformers or llama.cpp. Key models include:

LLaMA 3
Mistral-7B and other Mistral variations.
Command R and its enhanced versions.
Gemma models.
LLaMA 2 and Vicuna v1.3 models.

Key Features

Lightweight and High-Level: Kani streamlines the process of interfacing with language models by eliminating complex, opinionated prompt frameworks.
Model Agnostic: Users can switch backend models without major code changes, thanks to a straightforward implementation involving token counting and completion generation.
Automatic Chat Memory Management: Kani handles token history in chat sessions, freeing users from manual management.
Function Calling: This feature empowers language models to call functions using model feedback and retries, ensuring errors are intelligently managed.
Control Over Prompts: Users have total freedom in formatting prompts, unlike other frameworks that might impose strict structures.
Easy Iteration and Learning: Users only need to work with Python, simplifying the learning curve.
Asynchronous Design: Kani allows easy parallel execution of multiple chat sessions.

Installation

Kani requires Python 3.10 or higher. Depending on your needs, model-specific dependencies can be installed. Here's how you can set it up for different models:

# OpenAI models
$ pip install "kani[openai]"
# Hugging Face models
$ pip install "kani[huggingface]" torch
# Install everything
$ pip install "kani[all]"

For the latest features and updates, install directly from the main branch:

$ pip install "kani[all] @ git+https://github.com/zhudotexe/kani.git@main"

Get Started

Kani offers a fast-track integration using OpenAI engines but supports multiple models:

$ pip install "kani[openai]"

Then create a simple chatbot:

import asyncio
from kani import Kani, chat_in_terminal
from kani.engines.openai import OpenAIEngine

api_key = "sk-..."
engine = OpenAIEngine(api_key, model="gpt-4o-mini")
ai = Kani(engine)
chat_in_terminal(ai)
async def main():
    resp = await ai.chat_round("What is the airspeed velocity of an unladen swallow?")
    print(resp.text)
asyncio.run(main())

Function Calling

Easily expose Python functions to models with the @ai_function decorator:

from kani import AIParam, Kani, ai_function

class MyKani(Kani):
    @ai_function()
    def get_weather(self, location: Annotated[str, AIParam(desc="The city and state")]):
        return f"Weather in {location}: Sunny, 72 degrees fahrenheit."

Streaming and Customization

Kani's functionality extends to streaming responses and allows in-depth customizations without breaking code with its asynchronous design and integration flexibility.

Why Choose Kani?

Unlike other frameworks, Kani focuses on simplicity and flexibility without hidden processing of developer inputs, making it akin to Flask's simplicity as compared to Django's complexity. It's perfect for academic, professional, or hobby use without underlying code manipulations.

Concluding Remarks

Kani was developed by a team of PhD students at the University of Pennsylvania, aiming to contribute to the evolving field of NLP with an open and straightforward approach. It’s supported by several notable research bodies and actively maintained and used by its creators in ongoing research projects.