Solo Performance Prompting (SPP) Project Overview
Solo Performance Prompting (SPP) is a pioneering project that explores the amazing potential of large language models. The goal is to show how these models can solve tasks more effectively by emulating multiple personas working together within a single system, enhancing cognitive synergy. This exploration was detailed in a research paper, accepted at the prestigious NAACL2024 conference.
Exciting Updates
- May 8, 2024: Updated results with GPT-3.5 and LLama2 have been added, illustrating their powerful collaborative abilities.
- March 15, 2024: The research surrounding SPP has gained recognition, being accepted as a main paper at the NAACL2024 conference.
Getting Started with SPP
To dive into Solo Performance Prompting, follow these steps:
- Install Needed Tools: Run the command
pip install -r requirements.txtto install all necessary dependencies. - Configure API Access: Set up OpenAI API configurations using
config_template.shand execute it by runningsource config_template.sh. Note that for this project, Azure's API is utilized.
Quick Exploration
SPP offers quickstart scripts for engaging in different tasks, helping users familiarize themselves with the system's capabilities:
- Trivia Creative Writing: Run it by executing
bash scripts/trivia_creative_writing.sh. - Codenames Collaborative: Start this task with
bash scripts/codenames_collaborative.sh. - Logic Grid Puzzle: Get started with
bash scripts/logic_grid_puzzle.sh.
Resources for Exploration
- Prompts: All the prompts utilized in this project are stored in the
prompts/folder. - Data Sets: Data used in various tasks can be found in the
data/folder.
Understanding Paper Experiment Results
Results from experiments conducted for the paper are logged in the logs/ folder. These results showcase the performance of different models and configurations:
- GPT-4 Results: Found in directories like
gpt4_w_sys_mesandgpt4_wo_sys_mes, corresponding to the findings in Table 2. - GPT-3.5 and LLama2 Results: Their outcomes, akin to those showcased in Figure 6, are also available, demonstrating the effectiveness of the best solution configurations derived from GPT-4 experiments.
Details in Log Files
Logs contain comprehensive details such as:
- Evaluation Metrics: Real-world performance indicators like correct answers.
- Prompts: Full initial API prompts are logged.
- Responses and Parsing: Captures raw API responses and indicates whether they were successfully parsed.
- Processed Output: This parsed response helps evaluate task performance.
- Usage Information: Logs token numbers and costs incurred.
- Other technical details include model type, method, and settings like temperature.
Citing This Work
For those who find SPP beneficial or intriguing, the project encourages citations using the following reference:
@article{wang2023unleashing,
title={Unleashing Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration},
author={Wang, Zhenhailong and Mao, Shaoguang and Wu, Wenshan and Ge, Tao and Wei, Furu and Ji, Heng},
journal={arXiv preprint arXiv:2307.05300},
year={2023}
}
Acknowledgements
The SPP project was inspired by the structure of the Tree-of-thought official repository, showcasing gratitude towards its authors for providing a solid foundation through open-source efforts.
This overview provides a clear pathway for enthusiasts and researchers to engage with the innovative Solo Performance Prompting project, explore its potential, and contribute to its growing body of knowledge.