The Word Before the Word

Here is a trick. Read the next line and try very hard not to finish it.

peanut butter and …

Too late.

Maybe you supplied jelly. Maybe jam. Maybe sandwich (I do every time I read it). Maybe, if you grew up somewhere more sensible, neither. The point is not that there is a correct answer. The point is that a phrase which has not finished speaking has already built a small room inside your head, and a few words are standing in it, waiting.

That is the peculiar pleasure of language. It lets us be slightly ahead of ourselves.

Claude Shannon noticed this long before predictive text became an irritating rectangle above a keyboard. In 1951 he sat people down with a passage of English and asked them to guess it letter by letter, telling them only whether each guess was right. People were good at it. Good enough that Shannon could work backwards from their guessing to an estimate of how much information English actually carries: somewhere around one bit per letter, against the roughly 4.7 bits you would need if letters were shuffled independently. English is not a bag of independently drawn symbols. It is full of habits, promises, and small gravitational pulls.

You do this constantly, without noticing. Read “Once upon a …” and you already know the register, the rough shape of what follows, and that it is not a tax form. Read “Dear Sir or …” and there is essentially one candidate, because that phrase has exactly one habit. None of this is a party trick. It is most of what makes reading fast: you are not decoding a sentence so much as confirming a guess you already made.

Try a cleaner case. The astronomer adjusted the …

Something arrived. Probably telescope. Not because the sentence contains a rule about astronomers, and not because telescope was waiting at the end like a prize in a vending machine, but because that one word is simply brighter than the alternatives the sentence has left standing. Tax return is perfectly grammatical there and would be funny, which is the same fact stated twice. Elevator is possible, which is why prediction is a distribution and not a fortune teller.

I have a name for that brightness: the shadow of a sentence.

A sentence casts a shadow forward, made of things it has not said but has made easier to imagine. “The magician pulled a rabbit from the …” casts one kind of shadow. “The compiler reported a missing …” casts another. A good punchline works by letting you grow fond of one shadow and then switching off the lights.

This is not a grand new theory of linguistics. It is a way of noticing something ordinary: words do not arrive alone. They arrive with a small crowd of expectations, and the crowd does a surprising amount of the work.

Putting a number on surprise

Shannon’s real contribution was to give that feeling a unit.

If a word is likely in context, it carries little surprise. If it is unlikely, it forces a sharper revision of the picture you were building. The measure is called surprisal:

Notice what the formula does not contain: the word itself. No word is inherently surprising. Telescope is not a five-bit word or a two-bit word. After “the astronomer adjusted the,” it is nearly free. After “I found it in my soup,” it costs a great deal and does a completely different job. Surprisal is a property of a word and a context, never of a word alone.

It is also not a score for beauty, truth, or intelligence. A random word is surprising. So is a typo. So is a well-placed lie. The number only tells you how much the odds moved.

That same quantity is the thing every language model is trained to make small. The loss under next-token prediction is cross-entropy:

1
H(p, q) = -\sum_{x} p(x) \log q(x)

Here p is the true distribution over the next word and q is the model’s guess. During training p puts all of its weight on the single word that actually came next, so the sum collapses to one term: -log q(w_actual). That is the surprisal of the real next word, scored against the model’s own expectations. Average it over a corpus and you have the number a loss curve is tracking. Exponentiate that average and you have perplexity, the figure people quote when they compare models. Most training code works in natural log rather than log base 2, so you will see e^H more often than 2^H. Same measurement, different unit.

Two things worth saying out loud, because the number gets waved around a lot. First, a lower perplexity does not mean a model is smarter or more honest; it means the model is, on average, less surprised by text that resembles what it was trained on. Second, perplexities are only comparable when the tokenization and the evaluation text are the same. A model that chops words into smaller pieces gets more, easier predictions per sentence, and its number will look better for reasons that have nothing to do with understanding. Jurafsky and Martin’s chapter on n-gram models has the derivation, and the caveats, in full.

The same equation turned out to be useful in psycholinguistics, for a related reason: reading is incremental. You do not wait for the full stop to begin understanding; you build an interpretation while the sentence is still moving. John Hale made this precise in 2001, treating processing difficulty as a function of how much of your expected structure an incoming word rules out. Roger Levy sharpened it in 2008 into the claim that reading time is linear in surprisal, which is a strong, falsifiable, and reasonably well-supported thing to say about a brain.

The horse that makes everyone stumble

Read this once, at normal speed.

The horse raced past the barn fell.

It feels broken. Then, with a faint internal clunk, it repairs itself:

The horse that was raced past the barn fell.

The original is grammatical. So was your first reading, right up until fell arrived and found its job already taken. You had promoted raced to the main verb, because in the overwhelming majority of English sentences that is exactly what it would be.

This example earns its fame because nothing rare happens in it. No difficult vocabulary, no obscure reference, no trick of punctuation. The trouble is entirely in the future you built too quickly. These are called garden-path sentences, after the idiom about being led up one. The human name for the experience is “hang on, what?”

Here is a second one, harder than it looks: The old man the boats.

The trap is that you read the old man as a noun phrase, because you have read it as one ten thousand times before. It is not one here. The old is the subject, the way the poor or the young can be a subject, and man is a verb meaning to staff something. Old people crew the boats. Every word is common, the grammar is legal, and nothing in the first three words warns you — which is precisely the point.

Efficient reading depends on committing early. Early commitment is also exactly how you get trapped. There is no version of this system that is both fast and safe.

One direction only

Your brain gets to cheat, and this is the part I find genuinely interesting.

It reparses. By the time you finish the horse sentence a second time, raced has been quietly demoted from main verb to participle and nobody had to announce the correction. The wrong interpretation cost you a few hundred milliseconds and left no trace.

A language model generating text one token at a time does not get that. It can attend freely over everything already in its context, but it cannot unsay. Once a token is emitted it becomes part of the prefix, it conditions everything after it, and there is no mechanism for the model to stop and say “actually, raced was a participle, disregard the last four tokens.” The only move available is to continue in a way that makes the mistake look intentional, which, if you have watched a model dig itself deeper into a confident wrong answer, you have seen happen in real time.

Training makes this worse in a specific, structural way. During training the model always predicts the next token given the correct prefix, never given its own earlier output. It is graded on a path it never has to walk. At generation time it walks on its own footing for the first time, and it has had no practice recovering from a slip. Researchers call this exposure bias, which is a mild name for a strange situation.

A fair amount of decoding machinery exists to buy back a little of what your brain does for free. Beam search keeps several candidate continuations alive at once, so that a locally attractive token does not quietly doom the whole sequence. Self-consistency and other best-of-n methods generate many complete answers and choose among them afterwards, which is reanalysis by brute force: you cannot retract a token, but you can throw the whole attempt away and keep a better one. Reasoning-then-answering is arguably the same instinct in better clothes — make the revisable thinking cheap and clearly separate, and only commit at the end.

One correction to a claim I have seen made too casually, including in my own earlier draft of this piece: speculative decoding is not part of that family. It uses a small model to draft tokens and a large one to verify them, and the tokens it discards are discarded to preserve the large model’s distribution exactly. It is a speed trick, not a second thought. The output is meant to be identical. Worth keeping straight, because the two look similar from a distance and mean opposite things.

Fluent is not the same as true

Language models live inside this mechanism completely. Their entire job is to place probability mass over what comes next. Give one “the astronomer adjusted the” and it does not consult a dictionary entry for astronomers; it consults a vast learned map of how language tends to continue. Structurally, a language model is a machine for measuring the shadow of a sentence and then stepping into it.

Which is why fluent text deserves a particular kind of attention.

The most dangerous sentence is rarely the obviously wrong one. Obvious wrongness is high-surprisal; you catch it because it jolts. The dangerous sentence is the one that arrives smoothly enough that you do not notice it smuggled in an assumption. Right rhythm, right nouns, the shape of a sentence you were about to write yourself. The shadow does half the persuading before the claim has earned any of it.

A model minimizes surprisal against its training distribution, not against reality. Fluency and truth are correlated, mostly because most of the fluent text a model has ever read was also true, or was at least written by someone who believed it. Correlated is not identical, and the gap has a shape. A model does not fall silent when it does not know something. It produces the least surprising sentence available, because least surprising is the only thing it was ever asked for. Confidence is not a signal it emits on purpose. It is the default texture of the output.

A technical warning belongs here too, since the fit between these models and human minds is easy to overstate. Better next-token prediction does not automatically mean more human. Oh and Schuler found that larger Transformer models with lower perplexity often gave worse fits to human reading times: past a certain point, the models become good at predicting things people do not anticipate, and their surprisals stop lining up with our stumbles. The link between predictive models and human language is real, useful, and nowhere near settled.

Measuring it yourself

Everything above is conceptual: an invented astronomer, a hypothetical probability, a table I computed with a logarithm. If you want real surprisal numbers from a real model on a sentence of your choosing, it is about fifteen lines.

1
import math
2
import torch
3
from transformers import AutoTokenizer, AutoModelForCausalLM
4

5
tokenizer = AutoTokenizer.from_pretrained("gpt2")
6
model = AutoModelForCausalLM.from_pretrained("gpt2").eval()
7

8
def surprisal_per_token(sentence: str):
9
    ids = tokenizer(sentence, return_tensors="pt").input_ids
10
    with torch.no_grad():
11
        log_probs = torch.log_softmax(model(ids).logits, dim=-1)
12
    out = []
13
    for i in range(1, ids.shape[1]):
14
        lp = log_probs[0, i - 1, ids[0, i]].item()
15
        out.append((tokenizer.decode(ids[0, i]), -lp / math.log(2)))
16
    return out
17

18
for token, bits in surprisal_per_token("The horse raced past the barn fell."):
19
    print(f"{token!r:>10}  {bits:.2f} bits")

output:

' horse'  13.86 bits
' raced'  10.93 bits
' past'  5.53 bits
' the'  1.91 bits
' barn'  6.16 bits
' fell'  18.58 bits
    '.'  6.45 bits

gpt2 is the smallest public checkpoint, a few hundred megabytes, and it runs fine on a CPU one sentence at a time. Two caveats. The first token has no context, so it gets no score. And what I have been calling words, the model sees as tokens; surprisal itself is three or four of them to a GPT-style tokenizer. If you want a per-word number, sum the sub-word tokens that make it up.

Run it on the horse sentence and fell comes out well above everything before it, for precisely the reason it made you stumble. It is a small, cheap, slightly eerie thing to watch a number agree with your own reading.

The kindness of redundancy

All of this makes prediction sound like a case for ruthless efficiency. If the listener can guess the next part, why say it?

Because language is built for people who miss things.

We mishear. We forget the beginning of long sentences. We join conversations halfway through, or read them while thinking about dinner. Shannon’s own estimate put English at something like fifty percent redundant, and that redundancy is not waste. It is a handrail.

“Meet me by the north entrance, the side facing the river” is more robust than “meet me at the north entrance.” The second clause is technically unnecessary and does real work anyway: it lets the message survive a slightly wrong map in somebody else’s head.

There is a lovely result here that closes the loop with the horse. Speakers appear to spread information out, keeping the rate roughly even rather than letting it spike. Levy and Jaeger showed this with the optional word that: English lets you drop the relativizer in “the horse (that was) raced past the barn fell,” and speakers are measurably more likely to keep it exactly where what follows is hard to predict.

The redundancy appears exactly where the garden path would have been.

Good technical writing has the same courtesy. It says the important thing once, then lets it reappear in a usable form. Not louder, not three times in a row. Just before the idea becomes expensive to misunderstand.

Shannon’s paper is worth reading partly for this. It does not treat English as decoration wrapped around meaning. It treats it as a signal: structured enough to predict, redundant enough to survive.

The word after the word

When you write, it is tempting to ask only whether a sentence is correct.

Try the stranger question: what does this sentence make the reader expect next?

That one catches weak work early. If a paragraph ends by opening the wrong door, the next paragraph spends its first three lines dragging the reader back. If a term arrives before you have made room for it, it lands cold and stays cold. If a joke announces its own shadow too loudly, the room hears the punchline putting its shoes on.

The best writing does not eliminate prediction. It choreographs it. It makes a small promise, then keeps it, or breaks it at exactly the right moment.

Either way, the reader was already leaning forward. That was never the hard part.

Further reading

• Claude Shannon, Prediction and Entropy of Printed English (1951)
• John Hale, A Probabilistic Earley Parser as a Psycholinguistic Model (2001)
• Roger Levy, Expectation-based Syntactic Comprehension (2008)
• T. Florian Jaeger, Redundancy and Reduction: Speakers Manage Syntactic Information Density (2010)
• Byung-Doh Oh and William Schuler, Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times? (2023)
• Samy Bengio et al., Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks (2015), on exposure bias
• Muru Zhang et al., How Language Model Hallucinations Can Snowball (2023)
• Yaniv Leviathan, Matan Kalman and Yossi Matias, Fast Inference from Transformers via Speculative Decoding (2022)
• Daniel Jurafsky and James H. Martin, Speech and Language Processing, ch. 3: N-gram Language Models