vercel-llm-api

Introduction to the Python Vercel LLM API

The Python Vercel LLM API is a reverse-engineered software tool designed to wrap and interact with the Vercel AI Playground. This tool provides users with complimentary access to numerous large language models (LLMs), including well-known models like OpenAI's ChatGPT and others from providers such as Cohere, as well as various open-source models.

Features

The API offers several practical features that cater to different user needs:

• Model Access: Download and access a variety of language models effortlessly.
• Text Generation: Easily generate text by utilizing the available models.
• Chat Message Generation: Create conversational messages, enhancing interactive applications.
• Custom Parameters: Users have the flexibility to set specific parameters for model operations.
• Streaming Responses: Stream large responses to manage data efficiently as it is processed.

Limitations

While the API offers many advantages, it also has a few limitations users should consider:

• Hardcoded User-Agent: The user-agent is predefined and cannot be modified by users.
• No Authentication Support: The current version lacks support for user authentication, which could be important for some applications.
• Restricted Model Access: Some models, particularly those classified under "pro" or "hobby" tiers, are inaccessible due to usage policies.

Installation

Setting up the Python Vercel LLM API is straightforward. Users can simply run the following command to install the library:

pip3 install vercel-llm-api

Documentation

The library comes with a directory of examples, allowing users to dive right into the functionalities. By running the provided Python scripts, users can see how to operate the API in real-world scenarios:

python3 examples/generate.py

Using the Client

To start using the library, import it and create an instance of the client. If needed, a proxy can be specified for connection:

import vercel_ai
client = vercel_ai.Client()

For those requiring a proxy configuration:

import vercel_ai
client = vercel_ai.Client(proxy="socks5h://193.29.62.48:11003")

Downloading Models

Upon initialization, the client automatically downloads available models, storing them for easy access:

>>> print(json.dumps(client.models, indent=2))

Model IDs can also be retrieved:

>>> print(json.dumps(client.model_ids, indent=2))

Generating Text

Generating text with a model can be done using the generate function. Example of usage:

• Streamed Text Generation:

  for chunk in client.generate("openai:gpt-3.5-turbo", "Summarize the GNU GPL v3"):
      print(chunk, end="", flush=True)
  

• Non-Streamed Text Generation:

  result = ""
  for chunk in client.generate("openai:gpt-3.5-turbo", "Summarize the GNU GPL v3"):
      result += chunk
  print(result)
  

Generating Chat Messages

For generating chat messages, the chat function facilitates an interactive conversation model similar to OpenAI's API requirements:

messages = [
  {"role": "system", "content": "You are a helpful assistant."},
  {"role": "user", "content": "Who won the world series in 2020?"},
  {"role": "assistant", "content": "The Los Angeles Dodgers won the World Series in 2020."},
  {"role": "user", "content": "Where was it played?"}
]
for chunk in client.chat("openai:gpt-3.5-turbo", messages):
  print(chunk, end="", flush=True)
print()

Logging

To view debug messages or adjust the detail of logging, set the logging level appropriately:

import vercel_ai
import logging
vercel_ai.logger.setLevel(logging.INFO)

Copyright

The Python Vercel LLM API is licensed under the GNU General Public License v3. Developed by ading2210, this tool reflects a commitment to open-source software and community collaboration. All users have the rights to modify and share the software under the terms of this license, ensuring its continued evolution and adaptability.