22 Oct 2026

A surprise map of a book

A language model assigns a probability to every next byte. Its surprise at a byte, measured in bits, is the negative log of that probability: near zero when the text is predictable, high when something unusual appears. Plot it along a whole book and you get a map of where the text departs from what the model has learned.

We read Memoirs of the Court of Marie Antoinette, Volume 7 from the PG-19 test set, 164,000 bytes the model never saw in training, with CNet's 12-million-parameter complex-valued model. Born attention lets the model read text as a stream with a fixed-size state, so a book of any length costs it the same memory. The average was 1.22 bits per byte; a typical character cost 0.24 bits.

Two passages of the book with every character shaded by surprise; the typos remonstances, they, would and February,1820 stand out — Every character shaded by how surprised the model was; characters under 2 bits are left plain. The typing errors are the brightest marks.

Surprise in bits per byte along the book, with six numbered peaks — Average surprise over a sliding 512-byte window along the book. Numbered markers are the six most surprising passages.

What stands out

Dense runs of names. Five of the six peaks are lists of people: the royal family's attendants, the revolutionaries Hébert and Chaumette, a panel of physicians, the false claimants to the throne. Names are hard to predict byte by byte.
A change of structure. The sixth peak is the transcriber's index of quotations at the end of the file: sentence fragments in alphabetical order, not prose.
Typing errors in the source. These are the sharpest signals of all.

In the source text	What the model expected	Surprise
February,1820	a space after the comma	18.7 bits on “1”
remonstances	remonstrances	16.9 bits on “a”
they, would	no comma	8.2 bits on “,”
Laasus (Lassus a line later)	Lassus	8.0 bits on the second “a”

The most predictable stretch was plain narrative prose: the allied monarchs arriving in Paris, most characters well under one bit.

How it reads a whole book

A standard Transformer keeps a key and a value for every byte it has read, so its memory grows with the text. A model of the same size that kept this whole book in its attention cache would need about 2.7 GB by the last page. Born attention instead folds each byte into one fixed-size state: 12.8 MB on the first page, on the last page, and after five million bytes.

Standard attention stores one cache entry per byte and grows to 2.7 GB over the book; Born attention carries one fixed 12.8 MB state — Memory while reading the book. The 12.8 MB is measured; the Transformer line uses the matched model's measured cache, 67 MB per 4,096 bytes, which grows linearly.

Fixed memory is not the same as remembering everything. Today the model draws mostly on the last few thousand bytes, as we measured in Streaming is not memory; extending that is our next experiment.

Why this matters

A model that streams in fixed memory can make this pass over any amount of text, or any signal, cheaply: flagging transcription errors, unusual passages, or boilerplate in a corpus; ranking documents for training data; compressing. The same idea carries to radio, our first application, where streaming surprise over raw IQ samples is anomaly detection on the spectrum.

Method: CNet's CPU streaming decoder. To score on CPU quickly, the book was split into ten pieces scored in parallel, each warmed up on the 4 KB before it. The model draws mostly on the last few thousand bytes, so this closely matches a single unbroken pass. The code is in CNet (generate -eval ... -eval_dump).