Wandbot Project Introduction
Wandbot is a highly efficient question-answering bot, specially crafted to aid users in navigating the Weights & Biases documentation. It is designed to improve user interactions by providing accurate information swiftly, aiming to enhance the overall experience with Weights & Biases resources.
Key Features
Wandbot embodies several robust features to optimize its performance:
- Retrieval Augmented Generation: Utilizing a ChromaDB backend, wandbot ensures efficient and precise responses by fetching the most relevant documents in response to user queries.
- Continuous Data Improvements: By periodically ingesting data and generating comprehensive reports, the bot continuously improves its performance and accuracy. You can access the latest ingestion reports for more insights.
- Seamless Integration: The bot is fully integrated with popular collaboration platforms like Discord and Slack, making it easy for teams to incorporate into their workflow.
- Performance Monitoring: With Weights & Biases Tables, wandbot's performance is tracked and analyzed continually, ensuring ongoing improvements.
- Adaptive Model Fallback: When faced with challenges, such as GPT-4's inability to generate responses, wandbot adapts by choosing appropriate fallback models, maintaining its service reliability.
- Comprehensive Evaluation Metrics: Wandbot evaluates its responses using mixed metrics such as retrieval accuracy and string similarity, ensuring the correctness of bot-generated answers.
- Custom System Prompt: The bot employs a uniquely designed system prompt, enhancing its ability to respond accurately to user queries.
What's New in Wandbot v1.2.0
The latest version, Wandbot v1.2.0, introduces significant upgrades:
- Parallel LLM Calls: By adopting the LECL, the bot now handles parallel LLM calls, improving efficiency.
- ChromaDB Integration: Enhanced metadata filtering and speed by transitioning to ChromaDB from FAISS.
- Optimized Query Enhancer: This feature now functions more effectively, requiring only a single LLM call.
- Modular RAG Pipeline: The workflow is divided into modules - query enhancement, retrieval, and response synthesis for better management.
- Parent Document Retrieval: By adding this functionality, the bot offers improved context in responses.
- Sub-query Answering: Capable of handling complex queries by answering sub-queries.
- API Restructuring: The API design is reorganized into separate routers for retrieval, database, and chat functions, enhancing clarity and functionality.
Evaluation and Performance
Wandbot's evolution is evident from its growing response accuracy across versions:
- Version 1.0.0: 53.8% (baseline)
- Version 1.1.0: 72.5% (significant improvement)
- Version 1.2.0: 81.6% (latest enhanced version)
For the Japanese language, version 1.2.0 achieved 56.3% accuracy, and further improvements were introduced in version 1.2.1, raising the accuracy to 71.9%.
Installation and Usage
For those interested in using wandbot, installation requires Python version >=3.10.0,<3.11 and poetry for dependency management. The installation process is straightforward:
git clone [email protected]:wandb/wandbot.git
pip install poetry
cd wandbot
poetry install --all-extras
Once the system is set up, you can begin data ingestion, which pulls information from related repositories to power the retrieval augmented generation pipeline:
poetry run python -m src.wandbot.ingestion
Furthermore, to run the question-answering bot, several environment variables must be set before launching the application, whether it's for API, Slack, or Discord environments. Evaluation processes are also in place to assess and improve the bot's performance continuously.
Conclusion
Wandbot represents a sophisticated tool tailored for efficient and precise information retrieval within Weights & Biases resources. Its integration capabilities and continuous evolution make it a valuable addition to any team or individual seeking seamless interaction with Weights & Biases documentation. The ongoing improvements and detailed evaluations ensure Wandbot remains at the forefront of question-answering bots.