Streaming is not memory
Attention that runs as a fixed-size recurrence lets a model read an input of any length, in time linear in its length and in memory that does not grow. Results for such models are often summarized as a context length. We think that conflates three different claims, and we tested each one separately on byte-level PG-19 books.
- Streaming: the model processes a very long window with a constant-size state.
- Quality preservation: its loss does not degrade when the state is carried across the whole window.
- Long-range use: information from far back actually improves its predictions.
What held
Our 12-million-parameter complex model read each 5,242,880-byte window of the PG-19 test set in one unbroken pass with a 12.8 MB state, scoring 1.312 bits per byte. Carrying the state for five million bytes did not degrade anything: streaming and quality preservation both held.
What did not
Long-range use did not. Resetting the state every 4,096 bytes instead cost only 0.012 bits per byte, and the cost followed a simple law exactly: a fixed price per reset, the cold start, with no sign of information from further back. The reason is by design. Each attention head forgets at a fixed rate, with half-lives of 22 to 709 bytes, which is what made the model train well in the first place.
Streaming over millions of bytes is a property of the method. A memory of millions of bytes is not, yet. Extending it means giving some heads slower or learned forgetting, which is the next experiment.
An honest comparison
We also compared against a standard Transformer of the same size on the same data. At a short training budget our complex design came out ahead; at a longer one the Transformer led. A fix to our framework's weight initialisation, found and made in a few days, closed nearly a third of that gap. The architecture has had no tuning yet, against years of tuning for the Transformer recipe. We report all of this, including what went against us, in the paper.