From Prediction to Generation

David I. Inouye

Dynamic Content · Last Updated: October 30, 2026

A next-token model becomes a generator through repeated feedback

Factorize → choose a token → append → repeat → evaluate.

We have an email-prefix model. Now we must choose a continuation and decide when to stop.

A sequence probability is a product of conditional probabilities

For a fixed prompt \(c\) and continuation tokens \(w_1,\ldots,w_L\):

\[p_\theta(w_{1:L}\mid c)=\prod_{\ell=1}^{L}p_\theta(w_\ell\mid c,w_{<\ell}).\]

Synthetic token example:

\[p_\theta(\text{report, today, EOS}\mid \text{please send the})\] \[=p_\theta(\text{report}\mid c)\;p_\theta(\text{today}\mid c,\text{report})\;p_\theta(\text{EOS}\mid c,\text{report,today}).\]

Taking negative logs turns the product into the sum used in next-token training.

Teacher forcing uses the true prefix—even when the model is wrong

Both model blocks remain fixed above a bottom prefix row. The model prefers slides at 80 percent over the observed report at 10 percent. Training has report from the data; generation samples slides.

The bottom prefix row grows in place: training appends observed report, while a feedback arrow appends sampled slides for generation. The probability bars are labeled as predictions at the original prefix.

Training prefixes are all known in advance → a causal transformer can process them together.

Four minutes. Advance once to append the new token at the end of each bottom prefix row; both model blocks stay fixed. Bars remain labeled as the distribution at the original prefix. This is a comparison of conditioning rules, not a claim that training runs these prefixes sequentially. Synthetic probabilities, not measured model outputs. The observed email is “please send the report …”. At the prefix “please send the”, report is the supervised target even though slides has much higher model probability. Cross-entropy penalizes the low probability assigned to report. The next training task conditions on “… the report” regardless of the model’s distribution; no sample or argmax is fed into that task. This use of observed preceding tokens is teacher forcing. Both prefixes are known before the forward pass; the causal transformer computes their output rows together. An RNN can also use teacher forcing but still has a recurrent hidden-state dependency: known inputs are necessary here, not sufficient for position parallelism in every architecture.

At generation there is no observed next token to insert. One categorical draw selects slides in the pictured example; it is not guaranteed to select the most probable token. Append the draw, then evaluate the next distribution. Parameters stay fixed. “True” means the observed training sequence, not a guarantee that its statements are factually true. This original redraw follows the ground-truth-versus-feedback teaching idea in the prior course’s teacher-forcing illustration. [Sources] source:vaswani-attention-2017; source:d2l-rnn.

Greedy choice takes the largest bar; sampling uses all the bars

Synthetic distribution report 0.50, slides 0.30, agenda 0.20. The same probabilities partition a unit interval. A uniform draw of 0.64 falls in the slides interval, whereas greedy selection chooses report.

Greedy: choose \(\arg\max_j p_j\). Categorical sampling: choose token \(j\) with probability \(p_j\).

Temperature changes how concentrated the choices are

For the vocabulary-logit column \(\boldsymbol z\) and temperature \(T>0\):

\[p_{T,j}=\frac{\exp(z_j/T)}{\sum_{k=1}^{v}\exp(z_k/T)}.\]

The same synthetic logits at temperatures 0.5, 1 and 2. Lower temperature concentrates probability on report; higher temperature flattens the three bars.

Same model and prefix. Different sampling distribution.

Top-k fixes the count; top-p sets a mass threshold

Sorted synthetic probabilities. For a concentrated distribution .80,.10,.06,.04, top-k with k=2 keeps two tokens while top-p=.75 keeps one. For .30,.28,.24,.18, top-k keeps two and top-p keeps three to reach cumulative mass .82.

Keep the selected tokens → renormalize their probabilities → sample.

Top-p (nucleus sampling) keeps the smallest highest-probability set reaching the threshold.

Four minutes. A–D are synthetic token labels, sorted by current probability. Gray bars are removed; teal bars remain eligible. Top-k keeps the k most probable tokens (with a deterministic tie rule in this illustration). Top-p keeps a variable count: sort probabilities, take the shortest prefix whose cumulative mass is at least p, including the token that crosses the threshold. Here p=.75 needs one token for the concentrated distribution and three for the spread-out one. The retained mass can exceed p. Divide each retained probability by retained mass; then draw categorically. Top-k does NOT sample uniformly among k tokens, and top-p does NOT require each token individually to have probability above p. Both remove tail choices without changing weights. They can be combined with temperature; specify the order, for example temperature then filtering then normalization then sampling. Real-library tie and minimum-token conventions can differ. [Sources] source:huggingface-generation-methods-2020, Top-K Sampling and Top-p (nucleus) sampling.

Generation samples, appends, and runs the model again

Initial three-token prefix stays at the bottom, with one model block above and next-token probabilities at the top.

Sample slides at the top and return it to the right end of the same bottom row.

With slides appended, the same fixed model recomputes probabilities for the longer prefix.

Sample today and append it to the same row after slides.

The same model processes the five-token prefix and predicts EOS with high probability.

Sample EOS and stop. The bottom prefix remains please send the slides today.

Stop on EOS, a new-token cap, or the model’s context limit.

Three minutes. Advance five times through six frames. The model block and initial three token boxes never move. First compute probabilities; then sample slides and append its box at the right. Run the same model on the longer prefix, sample today and append its box. Finally run again, sample EOS, and stop. Sampling/append frames retain the previous distribution and explicitly precede the next model pass. EOS is not displayed as text. All probabilities and draws are synthetic. The same parameters are used on every pass. Each next prefix depends on the preceding draw; unlike teacher-forced training, future inputs are not available together. The example uses plain categorical sampling; the next comparison slide collects alternative choice rules. In real execution KV caching can reuse earlier computation while preserving this logical dependency. A length cap is needed because EOS need not be selected; stop before exceeding context capacity in this teaching loop. [Sources] source:d2l-rnn; source:huggingface-kv-cache-4-57-1.

Can we change how a trained model generates?

We switch a trained model from greedy decoding to top-p sampling.

What must change?

A. The model’s learned weights

B. The rule for choosing the next token

C. The causal attention mask

B. No retraining needed. We sample from the retained tokens instead of always taking the largest probability.

The same trained model supports different generation choices

Choice	How it chooses the next token	Main control
Greedy	Take the largest probability	Deterministic local choice
Categorical sampling	Draw from the full distribution	Random variation
Temperature sampling	Rescale logits by \(1/T\), then sample	Concentration
Top-k sampling	Keep \(k\) highest-probability tokens, renormalize, sample	Fixed candidate count
Top-p sampling	Keep the smallest set reaching mass \(p\), renormalize, sample	Adaptive candidate count

Inference choices: same weights and training; different generated outputs.

Evaluate the model + decoding rule for the task. Temperature and filters can be combined.

Three minutes. These choices are made during inference; changing them does not require retraining the baseline model. Their consequences are not independent of the model’s learned distribution or the task, so avoid a claim that one rule universally improves performance. Generation quality, diversity, repetition and task success may change. Temperature is a distribution transformation that can precede a top-k or top-p filter, not a mutually exclusive architecture choice. Greedy with a fixed tie rule is deterministic for fixed logits; sampling introduces randomness. These are the procedures covered here, not an exhaustive decoding taxonomy.

Preserve the evidence boundary: the email RNN/bigram comparison remains measured, while this transformer’s continuation quality and paired-prefix diagnostics remain unmeasured. Synthetic distributions do not supply empirical quality claims. Comparable held-out log loss needs the same split and tokenization; useful suggestions require task-specific evidence, and low-temperature agreement is not factual support. Next, the systems section asks how the same model and generation process fit memory and latency budgets. [Sources] source:huggingface-generation-methods-2020; source:leviathan-speculative-decoding-2023; source:aeslc-2019; source:chen-smart-compose-2019.