#Evaluation

This resource provides an in-depth review of diverse evaluation methods for large language models (LLMs), covering aspects like natural language processing and reasoning abilities. It features academic papers and projects assessing the robustness, ethics, and trustworthiness of LLMs. Regular updates ensure the most recent insights, with an open invitation for contributions to further refine the survey.

This objective overview explores various evaluation methods for large language models, focusing on assessing their knowledge, alignment, and safety. It presents an extensive collection of papers and methods curated by Tianjin University's team and underscores the necessity for meticulous evaluations to mitigate potential risks such as data leaks. The survey aims to foster responsible AI development and maximize societal benefits through a structured approach.

SAGE offers cutting-edge spelling correction with sophisticated Transformer models for multiple languages. It simulates human error patterns through statistical and rule-based spelling corruption methods. The tool supports model testing and evaluation on benchmark datasets, establishing SAGE as a versatile resource for improving text accuracy. Recent advancements include a paper acceptance at EACL 2024, highlighting its effective methodologies and ongoing advancements in spelling correction technology.

This repository provides a diverse selection of papers and materials aimed at exploring reasoning skills in large language models such as GPT-3. It delves into techniques like fully supervised finetuning and prompting for better understanding and implementing logical reasoning in NLP frameworks. Surveys and position papers offer thorough insights and analyses, encouraging further exploration of reasoning strategies. Researchers and developers are invited to enhance this collection by sharing additional papers, supporting a collaborative effort to advance NLP models.

This framework enables research into learning from human feedback using methods like RLHF, supporting feedback simulation and automated evaluations. It offers reference implementations for developers and researchers, facilitating research into instruction-aligned models. The framework is compatible with multiple language models, including GPT-4, and focuses on simulation accuracy for improved model evaluation and development.

This repository provides a detailed overview of how to improve large language models for code through instruction tuning. It describes components and datasets that enhance models such as OctoCoder and OctoGeeX with a focus on instruction-based fine-tuning. Explore strategic data approaches, including refined datasets like CommitPackFT, and evaluation methods across different programming languages. Training insights for models like OctoCoder and SantaCoder deliver actionable steps for refining model features, allowing for replication, assessment, and extension of existing models to enhance instructional efficacy in coding.

This repository provides a detailed evaluation of 28 superpixel algorithms utilizing 5 datasets to assess visual quality, performance, and robustness. It acts as a supplemental resource for a comparison published in Computer Vision and Image Understanding, 2018. Key updates include Docker implementations and evaluations of average metrics. The repository allows for fair benchmarking by optimizing parameters on separate training sets, focusing on metrics such as Boundary Recall and Undersegmentation Error.

SEED-Bench offers a structured evaluation setup for multimodal large language models with 28K expertly annotated multiple-choice questions across 34 dimensions. Encompassing both text and image generation evaluations, it includes iterations like SEED-Bench-2 and SEED-Bench-2-Plus. Designed to assess model comprehension in complex text scenarios, SEED-Bench is a valuable resource for researchers and developers looking to compare and enhance model performance. Explore datasets and engage with the leaderboard now.

Access a comprehensive compilation of prominent foundation model leaderboards, along with vital development tools and evaluation groups. This resource assists in navigating challenges and rankings in domains such as text, image, and video. It provides benchmarks and operational insights valuable for developers and researchers. Participate in the community for contributions, suggestions, and discussions to keep informed of current trends in foundation model evaluation. Utilize our toolkit for effective leaderboard exploration and management.

WebArena provides a self-contained, configurable web setting tailored for developing and evaluating autonomous agents. With Python 3.10 compatibility, it delivers installation guides, walkthroughs, and comprehensive development resources. Users can set up custom prompts, engage in detailed evaluation processes, and access a wide array of reproducible experimental tools.

PromptBench provides a versatile platform for evaluating Large Language Models (LLMs) with tools for performance analysis, prompt engineering, and adversarial prompt simulation. It supports numerous datasets and models, encompassing both linguistic and multi-modal varieties. The library facilitates swift assessments and rigorous testing, drawing on efficient evaluation methods akin to IRT models for accurate performance forecasts on new data. Ideal for researchers focused on improving LLM robustness without exaggerated claims.