Fact Checking with Prompt Chaining
The Fact Checking with Prompt Chaining project demonstrates an innovative approach to verifying facts using a sequence of logical prompts. This technique leverages the capabilities of Language Learning Models (LLMs) to enhance the accuracy of information by critically evaluating the assumptions that underpin an initial response. Here is an overview of how the process unfolds:
How It Works
-
Query the LLM: A user begins by submitting a question to the chosen LLM.
-
Initial Answer Formulation: The LLM processes the query and generates an initial answer based on its existing knowledge and data synthesis capabilities.
-
Self-Interrogation: The LLM then examines the assumptions that contributed to forming the initial answer. This self-evaluation is crucial for understanding the foundation of its response.
-
Assumption Verification: One by one, the LLM checks the validity of each assumption. It seeks confirmation or refutation of these assumptions to ensure their truthfulness.
-
Refined Answer Generation: Finally, using the verified or revised information, the LLM produces a new answer that reflects a more accurate understanding of the question.
Running the Project
To utilize this fact-checking tool, users can execute the following command in their terminal:
python3 fact_checker.py 'insert question here'
It is important to enclose the question in quotes when passing it as a command-line argument to prevent errors.
Alternatively, users can explore the project within a more interactive environment by utilizing the provided fact_checker.ipynb notebook.
Example of the Process
Consider the example question: "What type of mammal lays the biggest eggs?"
-
Initial Answer: The biggest eggs laid by any mammal belong to the elephant.
-
Assumptions Made:
- Elephants are mammals.
- Mammals lay eggs.
- Egg sizes vary.
- Elephants lay larger eggs than other mammals.
-
Verification of Assumptions:
- The elephant is a mammal: TRUE
- Mammals lay eggs: FALSE—Most mammals give birth to live young.
- Eggs come in different sizes: TRUE
- Elephants lay bigger eggs than other mammals: FALSE—Elephants do not lay eggs.
-
New Answer: The question cannot be answered as posed because elephants do not lay eggs, and most mammals give birth to live young.
Credits
This proof of concept was created by Jasper, showcasing an efficient approach to improve the accuracy of LLM-generated content by scrutinizing and adjusting underlying assumptions.
This project exemplifies the potential of prompt chaining in enhancing not only fact-checking systems but also in improving the robustness of AI-driven answers across various applications.