Email autocomplete suggests the next few words while you write
Typed prefix: “please send the”
Possible continuations: “report” · “slides” · “agenda”
The writer supplies the message and accepts or ignores a suggestion.
We predict the next token, then repeat to suggest a short continuation.
Suggestions should fit the prefix and arrive quickly.
First: turn text into choices the model can predict.
Two minutes. A concrete use case is completing routine work emails, not generating an entire email from a broad instruction. Smart Compose is evidence that completion is useful in a real product; these slides do not reproduce its model, dataset or measured performance. The toy example will use “please send the report” so every representation and recurrence can be followed by hand. Other listed suggestions illustrate ambiguity; the six-token arithmetic vocabulary below is intentionally smaller. No inbox access, personal messages or trained-model results are used. Later, a slow suggestion supplies the motivation for inference systems. [Sources] source:chen-smart-compose-2019.
Token IDs identify choices without ordering their meanings
Synthetic vocabulary for the worked examples:
please send the report becomes the ID sequence \((1,2,3,4)\) .
IDs select categories; subtracting two IDs does not measure similarity.
A token can be a word, a word piece, punctuation, or a special marker.
Next: turn each ID into learned features.
About 90 seconds. BOS marks beginning and EOS marks end for our example; actual tokenizers may choose different conventions. The six-entry vocabulary is synthetic and intentionally tiny. A real word-level baseline fits its vocabulary only on training emails and reserves an unknown token. Subword vocabularies let common pieces compose many words; no tokenizer algorithm is needed now. Vocabulary size is v in equations and V in code-shaped tensors. [Sources] source:vaswani-attention-2017.
The sequence model preserves positions while changing features
Token IDs
[B,L]Discrete input tokens
Input sequence \(X\)
[B,L,d]Embedded rows \(\boldsymbol{x}_\ell^{\mathsf T}\)
Hidden sequence \(H\)
[B,L,h]Hidden-state rows \(\boldsymbol{h}_\ell^{\mathsf T}\)
Output sequence \(Y\)
[B,L,V]Logit rows \(\boldsymbol{y}_\ell^{\mathsf T}\)
Target IDs
[B,L]Next-token labels
For one sequence: \(\boldsymbol{x}_\ell=X[\ell,:]^{\mathsf T}\) , \(\boldsymbol{h}_\ell=H[\ell,:]^{\mathsf T}\) , \(\boldsymbol{y}_\ell=Y[\ell,:]^{\mathsf T}\) (columns ).
Padding is absent data: exclude padded targets from the loss.
About two minutes. X and Y denote whole input and output sequences. Without a batch axis they are matrices with token positions as rows; with a batch axis use [B,L,d] and [B,L,V]. Assume batch_first=True in code; torch.nn.Linear applies over the final feature axis and stores W with output features as rows, matching the mathematical map W x; batch-first arrays use X W^T. Square brackets denote code tensor shapes. B,L,V are code dimension labels corresponding to batch count, sequence length and vocabulary count; d and h are feature widths. Padding lets different lengths share a batch, but padded context must also be ignored by the model. The per-position representation may use only the context allowed by the task. This strip specifies an interface, not a claim that every architecture computes all positions in parallel. The head is the same learned linear map at every position. [Sources] source:d2l-rnn; source:vaswani-attention-2017.
A fixed window turns continuation into familiar supervised prediction
\[\boldsymbol{y}_\ell=f_\theta\!\left(
\begin{bmatrix}\boldsymbol{x}_{\ell-2}\\\boldsymbol{x}_{\ell-1}\\\boldsymbol{x}_\ell\end{bmatrix}\right)
\in\mathbb{R}^{v}\]
Its limitation: anything before the window is unavailable to the model.
About 2.5 minutes. Here f_theta includes familiar hidden linear layers, ReLU and the vocabulary head; softmax follows logits to produce probabilities. Concatenation gives 3d input features, preserving the order of the selected tokens. For prefixes shorter than the window, use a consistent left-padding convention. This is a concrete baseline, not an empirical finding about a trained network. Parameter sharing occurs in the embedding table and in reuse of the same predictor at each position. Increasing the window can help but changes input width and is still a fixed limit. [Sources] source:d2l-rnn.
Which change lets the model distinguish these contexts ?
“Send the report today. Please attach the …”
“Send the slides today. Please attach the …”
Which change permits different predictions from a three-token model?
A. Add more hidden units
B. Include earlier tokens
C. Train on more emails
B. Identical input windows produce the same predicted distribution for fixed parameters.
Three minutes including vote and reveal. The synthetic email prefixes name different objects earlier and end with the same three tokens, “Please attach the”. The question asks what permits different conditional predictions, not what guarantees helpful suggestions. A larger model or more training cannot recover which prefix occurred when its input is identical. Random sampling could produce different strings from the same distribution, which is not evidence of distinguishing the contexts. This directly motivates carrying earlier information forward. Exactly three choices; the title and setup do not reveal the answer.
