llm-gateway

LLM Gateway Project Introduction

🤔 What is LLM Gateway?

LLM Gateway is a powerful tool designed to facilitate interactions with third-party language model providers such as OpenAI and Cohere. Essentially, it acts as a gateway, managing the data exchanged between users and these providers securely and efficiently. A key feature of LLM Gateway is its ability to track all sent and received data through a PostgreSQL database, employing PII scrubbing heuristics to ensure private information is protected before any data is sent out.

OpenAI, for example, often uses content provided by users, such as prompts and images, to refine their products like ChatGPT. With LLM Gateway, users can safely engage with OpenAI’s tools, relying on the gateway to maintain privacy and security. Additionally, LLM Gateway mirrors the ChatGPT interface by utilizing OpenAI's /ChatCompletion endpoint, thereby maintaining the communication solely within the API.

📦 Supported Models

LLM Gateway supports a wide range of models from various providers, ensuring flexibility and broad utility. Here are some of the supported models:

• OpenAI: GPT 3.5 Turbo, GPT 3.5 Turbo 16k, GPT 4
• AI21 Labs: Jurassic-2 Ultra, Jurassic-2 Mid
• Amazon: Titan Text Lite, Titan Text Express, Titan Text Embeddings
• Anthropic: Claude 2.1, Claude 2.0, Claude 1.3, Claude Instant
• Cohere: Command, Command Light, Embed - English, Embed - Multilingual
• Meta: Llama-2-13b-chat, Llama-2-70b-chat

⚒️ Usage

To utilize LLM Gateway, users need to configure their environment with the necessary API keys for the chosen provider, such as OPENAI_API_KEY for OpenAI. Detailed setup instructions for various providers like Cohere, OpenAI, and AWS Bedrock are available in the README documentation.

API Usage

For those interacting with OpenAI, here's an example of how to use the /completion endpoint with cURL:

curl -X 'POST' \
  'http://<host>/api/openai/completion' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "temperature": 0,
  "prompt": "Tell me what is the meaning of life",
  "max_tokens": 50,
  "model": "text-davinci-003"
}'

For chat-style interactions using the /chat_completion endpoint:

curl -X 'POST' \
  'http://<host>/api/openai/chat_completion' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "messages": [
    {"role": "assistant", "content": "You are an intelligent assistant."},
    {"role": "user", "content": "create a healthy recipe"}
  ],
  "model": "gpt-3.5-turbo",
  "temperature": 0
}'

Python Integration

For developers preferring to use Python, LLM Gateway provides a convenient wrapper:

from llm_gateway.providers.openai import OpenAIWrapper

wrapper = OpenAIWrapper()
wrapper.send_openai_request(
    "Completion",
    "create",
    max_tokens=100,
    prompt="What is the meaning of life?",
    temperature=0,
    model="text-davinci-003",
)

🚀 Quick Start for Developers

Developers enthusiastic about contributing to or running the project can rely on tools like Poetry and Pyenv for dependency and environment management. Here’s a simplified guide to getting started:

• Backend Setup: This includes installing necessary dependencies using Poetry and Pyenv. Docker users can skip some initial setup steps.
• Docker Development Loop: Running the project with Docker involves commands to manage the backend and frontend efficiently, allowing developers to easily spin up the development environment or bring it down as needed.

LLM Gateway provides the flexibility and security developers and users need when interacting with powerful language models from various providers, ensuring a seamless, safe, and efficient experience.