Attention, Transformers, and Generation

ECE 57000 — October 30, 2026

David I. Inouye

Attention chooses content and allowed context

• Learned queries and keys score matches; row softmax weights the value vectors.
• A causal mask excludes future inputs before softmax; training can compute every allowed output together.
• Unmasked attention without position signals reorders its outputs when inputs are reordered.

Today: represent order explicitly, then assemble a complete transformer.

About two minutes. October 28 covered the full attention operation, soft lookup, a numerical weighted sum, score scaling, cross-attention and causal masking. The class ended with permutation equivariance and the report/slides sentence example. Equivariance means outputs reorder, not that the output array stays unchanged. Distinguish attention parallelism with known inputs from sequential generation of unknown future inputs. The fixed causal mask already breaks arbitrary permutation equivariance; explicit position signals provide location/offset information directly. Repeat the exact final sentence slide before this distinction. [Sources] source:vaswani-attention-2017.

Column vectors become transposed rows in a sequence matrix

\[X=\begin{bmatrix}\boldsymbol{x}_1^{\mathsf T}\\\vdots\\\boldsymbol{x}_L^{\mathsf T}\end{bmatrix},\qquad \boldsymbol{q}_\ell=W_Q\boldsymbol{x}_\ell,\qquad Q=XW_Q^{\mathsf T}\]

Likewise \(K=XW_K^{\mathsf T}\) and \(V=XW_V^{\mathsf T}\). Example: four tokens, three features per vector.

Three minutes. This is a notation recap in the October 30 composition and a reusable notation bridge in the canonical topic. Every bold lowercase vector is a column. X has L rows and d input features; W_Q and W_K are d_k by d, W_V is d_v by d. Here L=4 and d_k=d_v=3, while input width d can differ. The row identity is q_ell^T=x_ellT W_Q^T, and similarly for keys and values. Stacking those transposed column equations yields the matrix projections. This matches PyTorch’s output-by-input weight storage. No change to batch-first array layout, parameter count, or model computation. Positions here are numbered 1–4 for the illustration; zero-based code indexes remain unchanged. [Sources] source:vaswani-attention-2017, section 3.2, expressed in the course’s column-vector convention.

Each query–key inner product fills one score-matrix entry

\[S=\frac{QK^{\mathsf T}}{\sqrt{d_k}},\qquad A=\operatorname{softmax}_{\rm row}(S),\qquad a_{\ell j}=\frac{e^{s_{\ell j}}}{\sum_{r=1}^{L}e^{s_{\ell r}}}\]

Dot products give scores; row softmax gives weights.

Three minutes. Follow highlighted row 2 of Q and column 3 of K transpose to entry(2,3). The query and key vectors are 3 by 1, so q2 transpose k3 is a scalar. Every output row compares one query with all four keys. A has the same 4 by 4 shape, but its entries are normalized weights rather than raw inner products. Each weight depends on the other scores in its query row through the denominator. Scaling by sqrt3 precedes normalization in this example. The displayed formula omits masking to isolate the multiplication; causal attention uses row softmax(S+M). The value matrix has the same four source positions, but in general its feature width d_v need not equal d_k. The highlight selects value 3 after normalization, not query 3. Subtract the row maximum when implementing softmax for numerical stability. [Sources] source:vaswani-attention-2017, section 3.2.

A sequence needs order beyond an unordered collection

Same words, different order	Requested action
Send the report before the slides.	Report first, slides second.
Send the slides before the report.	Slides first, report second.

Focus on report: its relationship to before has changed.

Without a mask or positions, self-attention gives report the same output vector in both sentences—just in a different row.

We want token representations to reflect order-dependent relationships.

Two minutes. A content-identical token carries the same contextual output after a permutation when no positions or masks are used. The array still has an order, and an order-sensitive downstream operation could use that array order, so do not claim that every possible model containing attention is unable to distinguish these sentences. The limitation is that this self-attention operation does not contextualize a token differently because it moved earlier or later. We want the sequence architecture itself to use those relationships. [Sources] source:vaswani-attention-2017, section 3.5.

Which change makes position explicit?

A causal attention layer already blocks future inputs.

Which change directly supplies each token’s absolute position?

A. Add a position vector to its embedding

B. Add another identical causal mask

C. Increase the number of attention heads

A. The mask controls access; a position vector explicitly represents location.

Three minutes including voting. Connect the repeated sentence-order example to the explicit position mechanisms next. A causal mask can indirectly supply order information, but it does not directly provide the absolute index as a feature. More heads offer more learned matching patterns without supplying indices. The next slide distinguishes access from explicit position signals; avoid saying masked attention has no order information. [Sources] source:vaswani-attention-2017, section 3.5.

Causal masks restrict access; position signals describe location

A fixed causal mask breaks arbitrary permutation equivariance: reordering tokens changes which context they may read.

Mechanism	What it supplies
Causal mask	Which earlier/current inputs a query may access
Absolute position signal	Where a token occurs in the sequence
Relative position signal	How far apart two token positions are

The mask does not explicitly encode token indices or pairwise offsets.

Next: add position features, or rotate query/key features using their positions.

Two minutes. With fixed triangular M, permuting X changes the content but leaves the mask attached to array positions; generally M differs from PMP transpose, so the unmasked equivariance proof no longer applies. The mask supplies directional structure and can allow a model to infer some positional information indirectly. Do not claim masked models contain no order information or that explicit position encodings are mathematically necessary for every architecture. In our teaching variants, we supply absolute location or relative offsets explicitly, rather than relying on indirect inference from allowed prefixes. Additive positions and RoPE are two alternatives, not simultaneous requirements. [Sources] source:vaswani-attention-2017, sections 3.1 and 3.5; source:su-roformer-2021.

Additive positions combine token features and location features

Look up the token’s embedding, then add a vector describing its position:

\[\boldsymbol{x}_\ell=E[w_\ell,:]^{\mathsf T}+\boldsymbol{p}_\ell,\qquad \boldsymbol{x}_\ell,\boldsymbol{p}_\ell\in\mathbb{R}^{d}\]

Fixed sinusoids: \(p_{\ell,2i}=\sin(\ell\omega_i)\), \(p_{\ell,2i+1}=\cos(\ell\omega_i)\), \(\omega_i=10000^{-2i/d}\).

Pair index \(i=0,\ldots,d/2-1\); even width \(d\).

Same token “the” at positions 0 and 2: \(E[\text{the},:]^{\mathsf T}+\boldsymbol{p}_0\ne E[\text{the},:]^{\mathsf T}+\boldsymbol{p}_2\).

Four minutes. The token embedding represents which token; the position vector represents where that occurrence is. Adding them gives the model a different input for the same token at different positions, while keeping width d. The final line can be read as two occurrences in “the report: the slides”; this is an illustration of token IDs rather than a tokenizer claim. Since the token embeddings cancel, x_2 minus x_0 equals p_2 minus p_0. No invented token-embedding values are needed.

Use zero-based feature indexing. In the d=4 illustration, omega_0=1 and omega_1=0.01 radians per position; the slow pair changes little over the displayed range. Read all four curves at each vertical slice, in sine/cosine pair order, to form the printed vector. Values are rounded to two decimals. These are actual values from the original formula, not learned features or an arbitrary frequency schedule. Larger d supplies more frequencies, from fast to slow. A learned position table is an alternative: learn one d-dimensional row per supported position instead of using this formula. Fixed formulas can be evaluated outside training lengths, but quality there is not guaranteed. Bold p_ell here is a position vector, distinct from the earlier vocabulary-probability vector. [Sources] source:vaswani-attention-2017, section 3.5.

RoPE rotates query and key pairs by their positions

Split each query/key into 2-D feature pairs; rotate every pair by a position-dependent angle.

\[R(\alpha)=\begin{bmatrix}\cos\alpha&-\sin\alpha\\\sin\alpha&\cos\alpha\end{bmatrix},\qquad \widetilde{\boldsymbol{q}}_\ell^{(i)}=R(\ell\theta_i)\boldsymbol{q}_\ell^{(i)}\]

\[[\underbrace{q_{\ell,0},q_{\ell,1}}_{\text{pair 0: }\ell\theta_0},\quad \underbrace{q_{\ell,2},q_{\ell,3}}_{\text{pair 1: }\ell\theta_1},\quad\ldots]^{\mathsf T}, \qquad \theta_i=10000^{-2i/d_k}\]

Fast and slow pairs cover different distance scales. Keys use the same rotations.

Four minutes. Superscript (i) denotes a feature pair, not a power or layer. For even head width d_k there are d_k/2 disjoint pairs; each pair has its own angular rate. This is a block-diagonal rotation, not repeated rotations of one shared 2-D vector. The displayed base 10000 is the original schedule; models may use different bases, partial rotary dimensions or scaling. Fast pairs change sharply over nearby positions; slow pairs vary across larger spans. These are distance scales, not assigned semantic roles. Rotations preserve each pair’s length. The rotation acts as R times a column vector. Key pair i rotates by s theta_i, with the same rate as query pair i. Values are not rotated in this construction. The figure uses a deliberately simple 30-degree rate and identical unrotated \([1,0]^{\mathsf T}\) vectors, not a claim that 30 degrees is the formula’s rate. [Sources] source:su-roformer-2021, sections 3.2.1–3.2.2.

RoPE compares positions through angle differences

Moving both tokens one step rotates both arrows by the same extra angle: their angle difference stays the same.

\[s\theta_i-\ell\theta_i=(s-\ell)\theta_i,\qquad (\widetilde{\boldsymbol{q}}_\ell^{(i)})^{\mathsf T}\widetilde{\boldsymbol{k}}_s^{(i)} =(\boldsymbol{q}_\ell^{(i)})^{\mathsf T}R((s-\ell)\theta_i)\boldsymbol{k}_s^{(i)}\]

The full dot product adds the matches from all feature pairs.

Why use RoPE?	Practical consequence
Relative offset enters the query–key match directly	Same separation gives the same positional comparison, with content held fixed.
Fixed rotations, no learned position table	Few extra operations; no separate learned vector for every index.

Used in Llama 2: RoPE is itself a positional encoding, an alternative to adding absolute position vectors.

Three minutes. A dot product depends on lengths and angle between vectors. Rotating both by the same angle preserves both. Content vectors can already have different initial angles; adding position rotations changes their angle difference by (s−ell)theta_i. Do not say that the score depends only on distance: content matters, and deeper contextual content can change when the sequence shifts. The sign in the displayed matrix identity follows column-vector conventions. Summing this identity across all disjoint pairs gives the full head dot product.

Why it is attractive: relative offsets enter the matching operation explicitly, pairwise rotations are inexpensive and preserve norms, and a fixed formula avoids a finite learned position lookup table. Fixed additive sinusoids also avoid a learned table: that advantage is not unique to RoPE. Contrast the design with additive sinusoids, where content and position are summed before learned projections and no equally direct relative-offset identity is guaranteed for arbitrary projections. Llama 2 section 2.2 documents adoption. This example does not measure popularity or prove universal superiority; architecture and empirical performance determine the choice. Long-context use often changes frequencies and training; evaluability beyond training lengths is not reliable extrapolation. [Sources] source:su-roformer-2021, sections 3.2–3.3; source:touvron-llama2-2023, section 2.2.

Multiple heads learn different ways to combine context

\[Y=\operatorname{Concat}(Y_1,\ldots,Y_b)W_O^{\mathsf T}\]

Each head has its own query/key/value projections; every head obeys the allowed context.

Next: combine attention with residual paths, normalization and a token-wise MLP.

Four minutes. b denotes the number of heads, not batch size. Each output Y_i is L by d_v; concatenation gives L by b d_v and W_O has shape d by b d_v. Concatenate features, not token positions; output returns to L by d. A common choice sets d_k=d_v=d/b. Splitting a single learned wide projection is an implementation of per-head projections, not literally partitioning the original input before learning. Heads may learn different access patterns, but do not promise specific semantic roles or independent causal explanations. This is an attention sublayer, not yet a complete transformer. [Sources] source:vaswani-attention-2017.

Implementation detail: padding masks exclude absent tokens

Two messages share a batch:

	Input positions
Message A	please · send · the · report
Message B	send · report · <PAD> · <PAD>

• Attention key mask: padded positions cannot supply values to real queries.
• Loss mask: padded targets contribute no training loss.

A causal mask still applies within each real prefix.

Two minutes. Padding is absent data, not evidence about an email. Key masking and target-loss masking serve different computations; neither substitutes for the other. Outputs for padded queries can be computed and then ignored, or avoided by the implementation. Handle entirely padded rows safely rather than softmax of all negative infinity. Packing independent messages also needs document boundaries so one message cannot read another. [Sources] source:vaswani-attention-2017; source:d2l-rnn.

Building a Transformer

David I. Inouye

Dynamic Content · Last Updated: October 30, 2026

A transformer combines context mixing and feature transformation

Transform features → preserve a residual path → normalize → stack blocks.

Keep the same task: use an email prefix to predict its next token.

About 30 seconds. Attention, positions and multiple heads are already available. We now assemble a causal pre-norm teaching model using familiar linear layers and residual addition. The original Transformer used post-norm; this section deliberately chooses a different normalization placement. [Sources] source:vaswani-attention-2017; source:xiong-layernorm-transformer-2020.

Attention mixes positions; a shared MLP transforms each row

\[\operatorname{MLP}(H)=\operatorname{ReLU}(HW_1^{\mathsf T}+\boldsymbol{1}\boldsymbol{b}_1^{\mathsf T})W_2^{\mathsf T}+\boldsymbol{1}\boldsymbol{b}_2^{\mathsf T}\]

\[H\in\mathbb{R}^{L\times d},\quad W_1\in\mathbb{R}^{m\times d},\quad W_2\in\mathbb{R}^{d\times m}\]

Same parameters at every position; separate parameters in each block.

Three minutes. The column vector 1 has L entries; each outer product 1 b^T broadcasts a bias across positions. m is the intermediate feature width, commonly larger than d; m=4d is an example, not a requirement. ReLU is a familiar teaching choice. Attention can mix positions; this MLP cannot introduce a new cross-position dependency. Its input already contains contextual information. Sharing across positions differs from reusing identical parameters across depth: our stacked blocks have their own parameters. [Sources] source:vaswani-attention-2017, section 3.3.

A residual path adds a learned update without changing the shape

\[H_{\mathrm{out}}=H+F(H)\]

The attention output projection and the MLP’s last layer both return \(d\) features.

Two minutes. Reuse the residual idea from ResNet. Addition is elementwise, not concatenation. The bypass offers a direct path through the block; it does not promise stable gradients or a small update. If the branch is zero, the block is exactly the identity. All rows and features must align for addition. Multi-head concatenation is projected back to d before adding it. [Sources] source:he-resnet-2016; source:vaswani-attention-2017.

LayerNorm normalizes one token’s features at a time

Like BatchNorm: subtract a mean, divide by a standard deviation, then learn a scale and shift.

\[\mu_\ell=\frac1d\sum_{j=1}^{d}h_{\ell j},\qquad \sigma_\ell^2=\frac1d\sum_{j=1}^{d}(h_{\ell j}-\mu_\ell)^2\]

\[\operatorname{LN}(\boldsymbol{h}_\ell)=\boldsymbol{\gamma}\odot \frac{\boldsymbol{h}_\ell-\mu_\ell\boldsymbol{1}} {\sqrt{\sigma_\ell^2+\epsilon}}+\boldsymbol{\beta}\]

• BatchNorm (CNN): one channel across batch examples and spatial locations.
• LayerNorm: one token across its \(d\) features—the last axis of [B,L,d].

At evaluation, LayerNorm still uses that token’s statistics; BatchNorm usually uses running statistics.

Four minutes. Learned gamma and beta each have d entries and are shared across token positions. Contrast with the previously discussed BatchNorm: this LayerNorm uses no other examples or positions and has no running batch statistics. Epsilon is a small positive constant. Population variance divides by d. With column h=[1,3]^T, gamma=[1,1]^T, beta=[0,0]^T, and epsilon approaching zero, the normalized row approaches [-1,1]^T. After learned scaling/shift, the output need not have zero mean or unit variance. Both train and evaluation compute statistics from the current token row. PyTorch nn.LayerNorm(d) matches this choice; normalizing over [L,d] would mix positions and can leak future information. [Sources] source:pytorch-normalization-2-7, LayerNorm documentation and forward implementation; source:pytorch-batchnorm2d-2-7. BatchNorm is shown in the CNN layout already taught; in a fully connected layer it instead aggregates each feature across examples. Both methods use a learned per-feature/channel affine transform, but normalize over different groups. PyTorch BatchNorm defaults to tracking running statistics; training/evaluation details and variance estimators differ, so the modules are not otherwise identical.

A pre-norm block adds two shape-preserving updates

\[U=H+\operatorname{MHA}_{\mathrm{causal}}(\operatorname{LN}_1(H))\] \[H_{\mathrm{out}}=U+\operatorname{MLP}(\operatorname{LN}_2(U))\]

Pre-norm is the usual modern LLM default. The original Transformer used post-norm.

Pre-norm	Post-norm
\(H+F(\operatorname{LN}(H))\)	\(\operatorname{LN}(H+F(H))\)
Normalize before the branch.	Normalize after addition.

Pre-norm leaves the bypass untouched.

Four minutes. Follow each branch and bypass in the diagram. LN1 and LN2 have separate learned scale/shift parameters. MHA includes all Q/K/V projections, causal masking, head concatenation and output projection. Each intermediate is L by d; the MLP expands internally and returns to d. Dropout is omitted for clarity. This is our chosen pre-norm variant, not the original 2017 post-norm arrangement or an exact named production model. The full model needs a final normalization after the stack. F denotes either the attention or MLP branch. The slide calls pre-norm the usual modern LLM default, not a universal standard across all Transformer architectures: recent models also use other placements, including Peri-LN. Pre-norm has favorable initialization-gradient behavior in Xiong et al., which helps explain its use for deep stacks; do not claim that post-norm is obsolete or every model uses pre-norm. Llama 2 uses pre-normalization with RMSNorm, a different normalization operator, not exactly the LayerNorm shown here. The remembered original Add & Norm diagram performs the residual sum first and LayerNorm after it, not LayerNorm immediately before the plus operation. [Sources] source:xiong-layernorm-transformer-2020, Figure 1 and Table 1; source:vaswani-attention-2017; source:touvron-llama2-2023, section 2.2; source:kim-periln-2025, sections 1–2.

Stacked blocks turn token features into next-token logits

\[H^{(0)}=E[w_{0:L},:]+P,\qquad Y=\operatorname{LN}_{\mathrm{final}}(H^{(N)})W_{\mathrm{vocab}}^{\mathsf T}+\boldsymbol{1}\boldsymbol{c}^{\mathsf T}\]

\[H^{(N)}\in\mathbb{R}^{L\times d},\qquad W_{\mathrm{vocab}}\in\mathbb{R}^{v\times d},\qquad Y\in\mathbb{R}^{L\times v}\]

Four minutes. E is the learned vocabulary table, w_0:L is the code-style slice of L input IDs, and P stacks transposed position columns as rows. N denotes block count, preserving L for sequence length. Head output dimension v is vocabulary size (V in code), distinct from the earlier projected value matrix V. Apply softmax to each vocabulary-logit row and compare with the shifted target. Our illustrative model does not tie embedding/head weights. For the alternative RoPE variant use H0=E[w,:] without P, then rotate Q/K within every attention sublayer; retain embeddings, residuals, MLPs, final normalization and vocabulary head. Do not silently combine both positional schemes. Both variants still generate autoregressively. [Sources] source:vaswani-attention-2017; source:xiong-layernorm-transformer-2020; source:su-roformer-2021.

One length-\(L\) input solves \(L\) prefix-prediction problems

\[\mathcal{L}(\theta)=-\frac{1}{L}\sum_{\ell=0}^{L-1}\log p_\theta(w_{\ell+1}\mid w_0,\ldots,w_\ell).\]

Known inputs allow parallel work; the mask preserves each allowed prefix.

Four minutes. The same synthetic sequence used for RNN training supplies five inputs and five shifted targets. BOS is the start token and EOS is the final target. Output y_ell is a column of vocabulary logits; the model stacks its transpose in row ell. Softmax and cross-entropy act independently on each output row before averaging the losses. Explain the report prediction: the output at input “the” sees BOS please send the, but cannot see the following input “report,” even though that token is present elsewhere in the training array. Each row equals the last-row prediction from running only its allowed prefix, with matching positions and deterministic settings. Thus we solve L overlapping prefix problems in one shared pass, without separately running L growing prefixes. With dropout, independent runs can differ because of sampled masks; this does not change causal information access. Padding positions are excluded from the loss in variable-length batches. [Sources] source:vaswani-attention-2017, sections 3.1 and 5.1; source:d2l-rnn.

Backpropagation follows different dependency graphs

Transformer benefit: parallel work across positions in forward and backward computation.

Five minutes. The loss boxes abbreviate the vocabulary head and cross-entropy; embeddings, normalization, residuals and MLPs are omitted for readability. In the RNN, h_ell depends on h_(ell-1), so hidden states require a sequential recurrence even when all inputs are known. Backpropagation through time follows that chain in reverse and adds contributions from losses at all positions. Input projections or output heads can still be batched; “sequential” refers to the recurrent state dependency, not every operation.

In the transformer, every upper row reads only allowed rows from the PREVIOUS layer, not already-updated rows of the same layer. All upper rows can therefore be computed together with masked matrix operations. Backpropagation reverses these edges and accumulates contributions to earlier representations and shared parameters; orange attention edges highlight only the final row to avoid clutter. All other rows also contribute. It does not require an RNN-style token-by-token reverse recurrence. Layers remain dependent in depth; gradient reductions, attention work, memory traffic and hardware determine actual speed. This capacity to exploit parallel hardware is a major computational advantage for training, not a claim of a measured speedup for our untrained model. Generation still depends on unknown future input choices. The numerical probe checks both logits and parameter gradients against the sum of separate prefix losses. [Sources] source:vaswani-attention-2017, section 4; source:d2l-bptt.

Can one forward pass score all shifted targets?

The training batch contains each full email.

Which model change would let an early prediction read its future target?

A. Compute all causal output rows together

B. Normalize each token over its own features

C. Remove the causal attention mask

C. Parallel computation is valid when every path respects the allowed prefix.

Three minutes including voting. The misconception is that having all targets in memory necessarily leaks them, or that parallel outputs must communicate with each other. The causal dependency graph prevents future input access; row-wise LayerNorm and the shared MLP preserve that property. With observed prefixes all positions can be computed in parallel during training. At generation, later input tokens do not yet exist, so the sampled output must be fed back before the next step. The executable probe compares full causal logits with separate-prefix logits and verifies future-token perturbation invariance. This is a structural check, not measured model quality. [Sources] source:vaswani-attention-2017, section 3.1.

A causal decoder is the common decoder-only LLM architecture

Our email model is a causal decoder. Other tasks can use different information paths and heads.

Three minutes. Wrap up training before broadening the architecture family. The diagrams show attention sources, with residual paths, normalization and repeated depth omitted. Q/K/V boxes denote learned projections, not raw token IDs; embeddings also include the chosen position mechanism. An encoder lets every supplied position attend to every other supplied position. A Vision Transformer uses patch embeddings, unrestricted encoder attention, and an image-classification head (typically from a class-token representation). The causal decoder-only form used in many generative LLMs predicts the next token from an allowed prefix. Not every language model is decoder-only. The original 2017 Transformer was an encoder–decoder for sequence-to-sequence prediction, demonstrated on translation. An encoder–decoder contextualizes a source sequence, then generates a target using causal target self-attention plus cross-attention to the full encoded source. Source and target lengths can differ. In cross-attention the target supplies Q and the source supplies K/V. This brief inset introduces information flow, not a second block derivation or a translation experiment.

Shared parameters and a different dependency structure provide different context access: fixed windows omit distant inputs, RNNs compress them through a sequential state chain, and attention reads allowed earlier representations directly. Access is a capability, not evidence of better predictions or length generalization. Full dense attention still has quadratic pairwise work. The RNN/bigram comparison remains the measured evidence; this transformer requires evaluation under the same frozen protocol. Next, return to our causal decoder and turn its next-token distributions into a generated continuation. [Sources] source:vaswani-attention-2017; source:touvron-llama2-2023; source:d2l-rnn; source:dosovitskiy-vit-2020, section 3 and Figure 1.

From Prediction to Generation

David I. Inouye

Dynamic Content · Last Updated: October 30, 2026

A next-token model becomes a generator through repeated feedback

Factorize → choose a token → append → repeat → evaluate.

We have an email-prefix model. Now we must choose a continuation and decide when to stop.

Thirty seconds. This section changes how we use the model, not its learned parameters. The conditional distribution is available from the vocabulary head; generating a sequence also requires a decoding rule and stopping conditions.

A sequence probability is a product of conditional probabilities

For a fixed prompt \(c\) and continuation tokens \(w_1,\ldots,w_L\):

\[p_\theta(w_{1:L}\mid c)=\prod_{\ell=1}^{L}p_\theta(w_\ell\mid c,w_{<\ell}).\]

Synthetic token example:

\[p_\theta(\text{report, today, EOS}\mid \text{please send the})\] \[=p_\theta(\text{report}\mid c)\;p_\theta(\text{today}\mid c,\text{report})\;p_\theta(\text{EOS}\mid c,\text{report,today}).\]

Taking negative logs turns the product into the sum used in next-token training.

Three minutes. The probability chain rule needs no independence assumption. The model approximates the conditionals; a finite context window can restrict their inputs. Tokens shown as words are a synthetic vocabulary, not a claim about a production tokenizer. EOS is an end-of-sequence token; including it assigns probability to a complete variable-length continuation. Lowercase w denotes realized token IDs. Prefix c is fixed conditioning information. The same identity connects the earlier cross-entropy training objective to full sequences. [Sources] source:d2l-rnn.

Teacher forcing uses the true prefix—even when the model is wrong

Training prefixes are all known in advance → a causal transformer can process them together.

Four minutes. Advance once to append the new token at the end of each bottom prefix row; both model blocks stay fixed. Bars remain labeled as the distribution at the original prefix. This is a comparison of conditioning rules, not a claim that training runs these prefixes sequentially. Synthetic probabilities, not measured model outputs. The observed email is “please send the report …”. At the prefix “please send the”, report is the supervised target even though slides has much higher model probability. Cross-entropy penalizes the low probability assigned to report. The next training task conditions on “… the report” regardless of the model’s distribution; no sample or argmax is fed into that task. This use of observed preceding tokens is teacher forcing. Both prefixes are known before the forward pass; the causal transformer computes their output rows together. An RNN can also use teacher forcing but still has a recurrent hidden-state dependency: known inputs are necessary here, not sufficient for position parallelism in every architecture.

At generation there is no observed next token to insert. One categorical draw selects slides in the pictured example; it is not guaranteed to select the most probable token. Append the draw, then evaluate the next distribution. Parameters stay fixed. “True” means the observed training sequence, not a guarantee that its statements are factually true. This original redraw follows the ground-truth-versus-feedback teaching idea in the prior course’s teacher-forcing illustration. [Sources] source:vaswani-attention-2017; source:d2l-rnn.