multi-doc-chatbot - Engage with Multiple Documents Using LangChain and OpenAI

Project Introduction: Multi Document Reader and Chatbot

Overview

The Multi Document Reader and Chatbot project is designed to provide users with a tool for interacting with multiple document types using advanced natural language processing technologies. Developed by leveraging LangChain and OpenAI, this project is an evolving set of Python scripts that enable users to create and interact with chatbots capable of processing various document formats.

Features and Capabilities

The project consists of multiple scripts, each with increasing complexity and functionality:

Single Document Interaction (single-doc.py): This script allows users to interact with a single PDF document by sending the entire content to a language model (LLM) for processing. It provides a simple interface for engaging with one document at a time.
Handling Long Documents (single-long-doc.py): For longer PDF documents, this script improves efficiency by using embeddings and a vector store to filter and send only relevant information to the LLM. This approach enhances performance by handling larger texts without overwhelming the model.
Multi-Document Chatbot (multi-doc-chatbot.py): The most complex script in the project, it enables interaction with multiple documents in various formats such as PDF, DOCX, and TXT. Unlike earlier versions, this script maintains chat history and conversation context. It uses embeddings and vector stores to deliver relevant data to the LLM, providing a more comprehensive chat experience. Users can communicate with the chatbot through a terminal interface, using standard input and output, with the option to exit by pressing q.

Getting Started

To begin using the Multi Document Reader and Chatbot, users should follow these steps:

Clone the Repository: Start by cloning the project's GitHub repository and navigating to its directory:
```
git clone [email protected]:smaameri/multi-doc-chatbot.git
cd multi-doc-chatbot
```
Set Up a Virtual Environment: Create and activate a virtual environment, then install the necessary dependencies:
```
python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
```
Configure OpenAI API Key: Set up the environment file and input the OpenAI API key:
```
cp .env.example .env
```
Ensure your .env file contains the API key in the following format:
```
OPENAI_API_KEY=sk-your-key-here
```
Start Interaction: Launch the multi-document chatbot script and interact with the documents placed in the /docs folder:
```
python3 multi-doc-chatbot.py
```
Users can exit the chat by entering q.

Advancements and Future Possibilities

The current implementation offers a robust foundation for building document-reading chatbots, though it is still evolving. Users have noted that while the setup can yield impressive results, fine-tuning is needed for optimal performance. Considerations for further enhancement include refining prompt templates, exploring models with higher token limits, and developing more sophisticated agents to improve answer accuracy and context relevance.

For those interested in gaining deeper insights into how these scripts are built, a detailed article is available that walks through the concepts and implementation.

Screenshot of Multi Document Reader and Chatbot

In summary, the Multi Document Reader and Chatbot project provides a dynamic and expandable platform for experimenting with state-of-the-art NLP technologies, making it a valuable resource for developers and researchers alike.