openai-forward - Comprehensive Proxy Service for Language Models with Intelligent Caching

OpenAI Forward: An Efficient Forwarding Service for Large Language Models

OpenAI Forward is a project designed to optimize the forwarding service for large language models. It offers a range of features aimed at providing efficient and convenient model forwarding services. This project supports both local and cloud-based language models, such as LocalAI and OpenAI, ensuring comprehensive compatibility and functionality.

Key Features

Universal Forwarding: OpenAI Forward can forward nearly all types of requests, making it highly versatile.
Performance-Oriented: Leverages asynchronous performance using libraries like uvicorn, aiohttp, and asyncio.
Cache AI Predictions: Provides a caching mechanism for AI predictions to accelerate access and save costs.
User Traffic Control: Allows custom request rate and token rate management to handle user traffic efficiently.
Real-Time Response Logging: Enhances the observability of Language Learning Models (LLMs) by logging responses.
Custom API Keys: Supports the use of customized keys to replace original API keys for added flexibility.
Multi-Target Routing: Facilitates routing multiple service addresses to different endpoints under a single service.
IP Whitelist/Blacklist: Implements IP whitelist and blacklist functionality to control access.
Automatic Retries: Ensures stability by automatically retrying failed requests.
Quick Deployment: Can be deployed locally or on the cloud swiftly using pip or Docker.

News

The new version v0.7.0 introduces configuration management through a WebUI.
The project now supports gpt-1106.
The backend cache has been switched to a high-performance database backend, FlaxKV.

How to Use

Quick Start

Installation

To start using OpenAI Forward, you can install it via pip:

pip install openai-forward

If you want the web UI version:

pip install openai-forward[webui]

Running the Service

Start the service with the following command:

aifd run

For the version with a web UI, use:

aifd run --webui

Upon successful startup, you will receive console feedback indicating the service is ready to run.

Proxying OpenAI Models

The default option for aifd run is to act as a proxy for OpenAI's API. For example, if you're setting up a server at https://api.openai-forward.com, you can integrate it into your Python application like this:

from openai import OpenAI
client = OpenAI(
   base_url="https://api.openai-forward.com/v1", 
   api_key="sk-******"
)

Proxying Local Models

Ideal for use with projects like LocalAI. Set up a local proxy by configuring environment variables or .env files to forward requests correctly.

Proxying Any Cloud Model

For example, to proxy Google's Gemini Pro, adjust the environment configuration to route the requests through OpenAI Forward.

Configuration

Use aifd run --webui to access the configuration page, typically found at http://localhost:8001. Customize configurations by creating a .env file in your project directory.

Contributions

The project welcomes contributions via pull requests or issue reporting.

License

OpenAI Forward is distributed under an MIT License, allowing for broad usage and modification.

With OpenAI Forward, users can ensure their language model deployments are more efficient, flexible, and easy to manage.