awesome-llm-human-preference-datasets

Awesome Human Preference Datasets for LLM: Project Overview

Introduction

The "Awesome Human Preference Datasets for LLM" project provides a carefully curated collection of open-source datasets focusing on human preferences. These datasets are essential for tuning large language models (LLMs) through instruction-tuning, reinforcement learning from human feedback (RLHF), and for evaluation purposes. This collection serves as a resource for researchers and developers interested in enhancing the performance and alignment of language models by understanding human preferences.

Dataset Collection

The project offers a variety of datasets, each with unique characteristics and applications:

1. OpenAI WebGPT Comparisons

   • This dataset includes 20k comparisons, each featuring a question, two model answers, and human-rated preference scores. It is primarily used for training the reward model in OpenAI's WebGPT project.

2. OpenAI Summarization

   • Comprising 64k text summarization examples, this set includes both human-written and model-generated summaries, rated by humans. It's employed in research focusing on summarization through human feedback.

3. Anthropic Helpfulness and Harmlessness Dataset (HH-RLHF)

   • Featuring 170k human preference comparisons, this collection is vital for training models to be both helpful and harmless. It includes sub-datasets focused on different aspects of human feedback.

4. OpenAssistant Conversations Dataset (OASST1)

   • This comprehensive dataset includes 161k multilingual assistant-style messages curated and annotated by humans, contributing to a rich repository of conversation examples.

5. Stanford Human Preferences Dataset (SHP)

   • With 385k human preference records, this dataset primarily sourced from Reddit is used for refining RLHF reward models across 18 domains.

6. Reddit ELI5

   • This dataset provides 270k instances of questions and answers sourced from specific Q&A Reddit subforums, helping to understand nuanced human interpretations.

7. Human ChatGPT Comparison Corpus (HC3)

   • Containing human and ChatGPT responses to numerous questions, this dataset allows for comparative analysis and is also available in a Chinese variant.

8. HuggingFace H4 StackExchange Preference Dataset

   • Pulling from StackOverflow, this dataset includes millions of questions and corresponding answers, scored by user votes, to aid in training language models with community-driven feedback.

9. ShareGPT.com

   • Incorporating interactions from users with ChatGPT, this dataset, although currently limited due to API restrictions, serves as a reflection of varied user engagements.

10. Alpaca

    • Produced using OpenAI's advanced text generation engine, this dataset aids in self-instruct training by providing diverse instructional content.

11. GPT4All

    • Compiling 1M prompt-response examples generated via the GPT-3.5-Turbo API, this dataset supports robust training and evaluation methodologies.

12. Databricks Dolly Dataset

    • Featuring instructions developed by Databricks employees, this collection covers multiple question-answering formats and idea-generation tasks, demonstrating practical applications of LLMs.

13. HH_golden

    • Building on the Anthropic HH dataset, this set refines "harmless" responses using GPT4 to enhance the alignment of LLMs with human expectations.

Conclusion

The "Awesome Human Preference Datasets for LLM" project is an invaluable repository for researchers and developers working on language model alignment and refinement. It provides diverse datasets aimed at understanding and incorporating human preferences into LLMs, facilitating advancements in AI that are both human-centered and contextually aware.