#information retrieval

Baidu is China's foremost search engine, offering users swift access to information through an expansive database of over a trillion Chinese web pages.

MS MARCO Web Search provides a vast dataset with millions of real user query-document interactions from ClueWeb22's 10 billion pages, aiding in research tasks like embedding models and retrieval systems. This resource supports studies across 93 languages, fostering advancements in machine learning and information retrieval.

RAPTOR offers an advanced approach to language models with its recursive tree structure, improving the efficiency of information retrieval in large texts. It supports integration with custom models for summarization and question-answering, making it highly adaptable to different research requirements. The open-source nature encourages continuous enhancement through community contributions.

RAGatouille connects the latest research with practical RAG pipeline practices, boosting the ease of use and modularity of retrieval methods. The platform leverages models like ColBERT to enable enhanced generalization capabilities, data efficiency, and the ability to train in non-English languages. It simplifies the integration of sophisticated retrieval techniques without deep diving into complex literature, making it clear and accessible. Users can use, train, and fine-tune retrieval models seamlessly within diverse RAG scenarios, thanks to the robust defaults and customizable elements of RAGatouille. This focus on user experience is supported by comprehensive integration options with top frameworks such as Vespa, LangChain, and Intel's FastRAG, offering flexibility for smooth deployment and scaling.

Introducing an AI system that generates Wikipedia-like articles via extensive internet research and multi-perspective questioning. Suitable for academic and editorial use, it supports both individual and collaborative processes for effective knowledge curation, with over 70,000 users engaging in its feature previews.

SPLADE utilizes BERT to build sparse models that enhance the first-stage ranking in information retrieval tasks. With the adoption of sparse representations, the models achieve efficiency gains and clarity in lexical matching. Recent improvements include static pruning for neural retrievers and advanced training techniques. The models are versatile across various domains. Pre-trained versions are accessible on Hugging Face, allowing for efficient performance comparable to traditional methods, with reduced latency.