Knowledge sharing • 3 min read •
ATLAS: A New Solution for Long-Context Language Models
Google's new ATLAS architecture pushes beyond today's context limits. Its adaptive memory module learns even during inference, enabling stable performance on sequences of up to 10 million tokens. The result? Processing entire books, massive codebases, or even hours of video may soon become practical.
Today's large language models, such as GPT and Gemini, can process impressive amounts of text, but they still face a major limitation: the size of their context window. This determines how much information a model can "keep in mind" at once. Although modern systems can already handle hundreds of thousands—or even millions—of tokens, their performance typically degrades as the context grows longer. The main reason is the Transformer architecture itself, where every token attends to every other token, causing computational complexity to grow quadratically with sequence length.
Researchers have proposed several approaches to address this challenge. Windowed and sparse attention mechanisms reduce computation by considering only nearby or selected tokens. Modern Recurrent Neural Networks (RNNs) and State Space Models (SSMs) take a different approach by continuously updating an internal state while processing the input. While these methods improve efficiency, none has fully solved the challenge of handling contexts spanning several million tokens.
This is where ATLAS, introduced by Google researchers in 2025, comes in. ATLAS is the first member of a new family of architectures called Deep Transformers. Its key innovation is a memory module that can continue learning during inference, not only during training. As each new token arrives, the model predicts the information associated with it. Whenever the prediction is incorrect, the memory module immediately updates itself, continuously adapting to the current document while leaving the underlying model parameters unchanged.
The results are remarkable. On the BABILong benchmark, ATLAS maintained approximately 80% accuracy even when processing sequences of 10 million tokens, significantly outperforming previous approaches. This represents a major leap forward, as practical context limits had previously been around two million tokens. Although ATLAS has so far only been evaluated at relatively modest model scales, leaving open questions about its behavior with tens of billions of parameters, its potential is already clear.
The key idea behind ATLAS is that language models should not simply attend to larger contexts—they should learn and remember while solving the task itself. This opens the door to applications such as processing entire books, analyzing massive software repositories, or even understanding videos lasting several hours.
Sources:
