rags - Create RAG Pipelines with Ease Using Natural Language

RAGs Project Introduction

RAGs is an innovative Streamlit application designed to create a RAG (retrieval-augmented generation) pipeline from various data sources using natural language. This allows users to set up a system where they can retrieve and interact with specific information from their chosen datasets.

Key Features

RAGs allows users to perform the following tasks:

Task Description: Start by describing the task you wish to accomplish, such as “load a specific web page,” and specify the desired outputs from your RAG system, like retrieving a certain number of documents.
Configuration View: Access the configuration view to review and modify the generated parameters, such as the top-k retrieval criteria or summarization needs.
Data Query: Use natural language queries to interact with the RAG agent, asking questions and retrieving answers from your chosen data.

The project draws inspiration from OpenAI's GPTs to provide a similar experience tailored to your data.

Installation and Setup

To get started with RAGs:

Clone the project repository and navigate to the rags folder. It is recommended to create a virtual environment to manage dependencies:
```
python3 -m venv .venv
```
Install the necessary packages with:
```
poetry install --with dev
```
Make sure to configure OpenAI as the default agent. Add your OpenAI key in a .streamlit/secrets.toml file located in the home folder:
```
openai_key = "<openai_key>"
```
Launch the app from the home page file:
```
streamlit run 1_🏠_Home.py
```

Note: If you experience issues after upgrading RAGs, try deleting the cache folder in your home directory as there might have been changes in the stored data structure.

Detailed Overview

The application is organized into several sections, reflecting the setup process outlined above.

1. 🏠 Home Page

This is where you begin building your RAG pipeline by providing instructions to the "builder agent." Essential components include:

Dataset Description: Specify if you’re using a single local file or a webpage.
Task Description: Define the task in concrete terms, which helps initialize the system prompt for the LLM (Large Language Model).
Parameter Definition: Determine the configuration settings needed for your RAG setup.

2. ⚙️ RAG Config

In this section, the RAG system parameters are displayed. These parameters are generated by the builder agent based on the initial setup and can be manually edited. Some of the key parameters include:

System Prompt
Summarization Option: Decide whether to add summarization tools in addition to top-k retrieval.
Top-K Retrieval
Chunk Size
Embedding Model
LLM (Large Language Model)

After making any changes, the "Update Agent" button allows you to implement the new settings.

3. Generated RAG Agent

Once your RAG agent is set up, this section provides a chatbot interface where you can pose questions to the RAG agent. The agent picks the most relevant tools, such as top-k vector search or summarization, to answer your queries effectively.

Supported LLMs and Embeddings

Builder Agent

By default, RAGs uses OpenAI for constructing the builder agent, configurable via the core/builder_config.py file. Although customizable, GPT-4 variants are recommended for reliable agent creation.

Generated RAG Agent

Configurations can be established through both natural language and manual input.

LLMs: Supported models include:
- OpenAI (e.g. "openai:gpt-4-1106-preview")
- Anthropic (e.g. "anthropic:claude-2")
- Replicate
- HuggingFace (e.g. "local:BAAI/bge-small-en")
Embeddings: Defaults to text-embedding-ada-002, with the option to use Hugging Face models (e.g., "local:BAAI/bge-small-en").

Resources

For further assistance, users can report issues on GitHub or join the community on Discord. The RAGs app was built leveraging the LlamaIndex Python library. For more insights, read the launch blog post here.