#Context Extension

LongRoPE extends the context window of large language models past 2 million tokens using non-uniform positional embeddings and a 256k fine-tuning strategy. This method sustains performance across various context lengths, supporting in-context learning and long document summarization.