#long-context

Explore a language model with 1.9 billion parameters and 32K context support, designed for efficient long-document processing. It excels in multilingual translation, particularly in East Asian languages, and offers role-playing capabilities. Recent updates include adaptations for llamacpp and Ollama, with open-source checkpoints available. Evaluation results demonstrate its competitiveness with larger models, making it ideal for chat, translation, and fine-tuning applications.

The Recurrent Memory Transformer (RMT) enhances AI model performance by using memory-augmented segment-level transformers. Designed for Hugging Face models, it incorporates special tokens to optimize memory and sequence processing. Features include comprehensive training examples, gradient accumulation, and metric management. It supports tasks with extensive context requirements and is developed in partnership with DeepPavlov.ai, AIRI, and London Institute for Mathematical Sciences.

BABILong evaluates NLP models on their ability to handle long documents filled with disparate facts, incorporating bAbI data and PG19 texts for diverse reasoning tasks. The benchmark's 20 tasks, including fact chaining and deduction, challenge even advanced models like GPT-4. Contributions to the benchmark are encouraged to further collective insights into LLM capabilities.