Sequence Prediction and Representation (continued)

ECE 57000 — October 26, 2026

David I. Inouye

Next-token prediction connects text, learned features and targets

• Classification summarizes an image; segmentation preserves a prediction at every location.
• Token IDs select rows of a learned embedding table; ID differences do not measure meaning.
• Each prefix predicts the following token, using only the context available at that position.

Today: finish the training examples, then compare fixed windows, recurrent memory and attention.

About two minutes. Recap October 23: ResNet pooling and spatial compression, U-Net spatial outputs and skip fusion, then email completion, token IDs, embeddings and shifted targets. Repeat the exact stopping slide before tensor shapes, held-out threads and context models. The examples remain synthetic. [Sources] source:d2l-rnn; source:chen-smart-compose-2019.

Each input position predicts the token immediately after it

\[\boldsymbol{p}_\ell=\operatorname{softmax}(\boldsymbol{y}_\ell), \qquad \mathcal{L}_\ell=-\log p_{\ell,w_{\ell+1}}\]

Output \(\boldsymbol{y}_\ell\) may depend on \(w_0,\ldots,w_\ell\): all previous tokens and the current token.

About 2.5 minutes. y_ell is a row containing v vocabulary logits; Y stacks these output rows. The discrete target remains w_(+1); p_ell is the deterministic vector of predicted conditional probabilities. w_ell is an observed integer token ID; p_{,w_{+1}} selects the probability assigned to the observed next ID. Reuse multiclass cross-entropy. With input positions 0 through L−1, targets use positions 1 through L. BOS lets the example predict its first ordinary token and EOS marks an ending. The target is available to compute loss, not to compute its own prediction. [Sources] source:d2l-rnn.

The sequence model preserves positions while changing features

Stage	Tensor shape	Meaning
Token IDs	[B,L]	Discrete input tokens
Input sequence \(X\)	[B,L,d]	Embedded rows \(\boldsymbol{x}_\ell\)
Hidden sequence \(H\)	[B,L,h]	Hidden-state rows \(\boldsymbol{h}_\ell\)
Output sequence \(Y\)	[B,L,V]	Logit rows \(\boldsymbol{y}_\ell\)
Target IDs	[B,L]	Next-token labels

For one sequence: \(\boldsymbol{x}_\ell=X[\ell,:]\), \(\boldsymbol{h}_\ell=H[\ell,:]\), \(\boldsymbol{y}_\ell=Y[\ell,:]\) (rows).

Padding is absent data: exclude padded targets from the loss.

About two minutes. X and Y denote whole input and output sequences. Without a batch axis they are matrices with token positions as rows; with a batch axis use [B,L,d] and [B,L,V]. Assume batch_first=True in code; torch.nn.Linear applies over the final feature axis and stores its weight transposed relative to the mathematical right-multiplication matrices here. Square brackets denote code tensor shapes. B,L,V are code dimension labels corresponding to batch count, sequence length and vocabulary count; d and h are feature widths. Padding lets different lengths share a batch, but padded context must also be ignored by the model. The per-position representation may use only the context allowed by the task. This strip specifies an interface, not a claim that every architecture computes all positions in parallel. The head is the same learned linear map at every position. [Sources] source:d2l-rnn; source:vaswani-attention-2017.

Test completion on held-out email threads

Group and split threads	Then create examples
Training threads	Training prefixes and next-token targets
Validation threads	Validation prefixes and next-token targets
Reserved test threads	Final held-out prefixes and targets

• Overlapping windows from one email share much of their text.
• Fit the vocabulary on training emails only; handle unseen tokens explicitly.

Now build a baseline: how much of the prefix should it read?

Two minutes. These are design rules for email experiments. The later AESLC experiment uses normalized-subject and duplicate-prefix groups as a proxy because authoritative thread IDs are unavailable; it does not verify true thread independence. Split whole email threads before making overlapping windows, fit vocabulary only on training text, and keep test threads reserved. Group by thread to keep quoted replies and near-duplicates together; audit duplicates across groups. If the intended use is future mail or new writers, choose a time- or writer-based split accordingly. Reset recurrent state between independent messages. The synthetic examples here explain the mechanics. The later held-out AESLC results measure token prediction, not product usefulness. Next we can build the simplest model under this setup. [Sources] source:chen-smart-compose-2019.

A fixed window turns continuation into familiar supervised prediction

\[\boldsymbol{y}_\ell=f_\theta\!\left( [\boldsymbol{x}_{\ell-2}\;\boldsymbol{x}_{\ell-1}\;\boldsymbol{x}_\ell]\right) \in\mathbb{R}^{v}\]

Its limitation: anything before the window is unavailable to the model.

About 2.5 minutes. Here f_theta includes familiar hidden linear layers, ReLU and the vocabulary head; softmax follows logits to produce probabilities. Concatenation gives 3d input features, preserving the order of the selected tokens. For prefixes shorter than the window, use a consistent left-padding convention. This is a concrete baseline, not an empirical finding about a trained network. Parameter sharing occurs in the embedding table and in reuse of the same predictor at each position. Increasing the window can help but changes input width and is still a fixed limit. [Sources] source:d2l-rnn.

Which change lets the model distinguish these contexts?

“Send the report today. Please attach the …”

“Send the slides today. Please attach the …”

Which change permits different predictions from a three-token model?

A. Add more hidden units

B. Include earlier tokens

C. Train on more emails

B. Identical input windows produce the same predicted distribution for fixed parameters.

Three minutes including vote and reveal. The synthetic email prefixes name different objects earlier and end with the same three tokens, “Please attach the”. The question asks what permits different conditional predictions, not what guarantees helpful suggestions. A larger model or more training cannot recover which prefix occurred when its input is identical. Random sampling could produce different strings from the same distribution, which is not evidence of distinguishing the contexts. This directly motivates carrying earlier information forward. Exactly three choices; the title and setup do not reveal the answer.

A recurrent state carries earlier information into the next step

Fixed-window predictor	Recurrent predictor
Reads the most recent \(k\) tokens	Reads the current token and previous state
Earlier tokens fall outside the input	Earlier tokens can affect the state
Input width depends on \(k\)	State width stays fixed

Next question: how can one learned update carry useful earlier information forward?

About 90 seconds. Introduce recurrence as a possible mechanism, not proof that useful information will be retained. A fixed-width state is a learned summary, not the complete stored history. Long sequences still require more processing. We have defined the task, built training examples, and found the fixed-window limitation. The next topic adds a learned state and shows how to train its update. [Sources] source:d2l-rnn.

Recurrent Neural Networks (RNN)

David I. Inouye

Dynamic Content · Last Updated: October 26, 2026

One transition combines the current token and the previous state

State update

\[\boldsymbol{h}_\ell=f_\theta(\boldsymbol{x}_\ell,\boldsymbol{h}_{\ell-1})\]

Vocabulary head

\[\boldsymbol{y}_\ell=g_\phi(\boldsymbol{h}_\ell)\]

The state changes with the prefix. The parameters \(\theta,\phi\) are shared.

Next: choose a concrete \(f_\theta\) and \(g_\phi\).

Two minutes. Resume the running email-completion task. The same representation has two consumers: the next state update and the current vocabulary head. h_0 is an initial zero vector for a new email in this example; it is not a new learned parameter for each document. The hidden state changes with the prefix even though the transition parameters stay fixed. [Sources] source:d2l-rnn.

Familiar layers implement the recurrent update

\[\boldsymbol{h}_\ell=\operatorname{ReLU}\!\left( \boldsymbol{x}_{\ell}W_x+\boldsymbol{h}_{\ell-1}W_h+\boldsymbol{b}\right)\]

\[\boldsymbol{y}_\ell=\boldsymbol{h}_{\ell}W_y+\boldsymbol{c}, \qquad\boldsymbol{p}_\ell=\operatorname{softmax}(\boldsymbol{y}_\ell)\]

Embedding and state	Learned matrices	Vocabulary output
\(\boldsymbol{x}_\ell\in\mathbb{R}^{d}\)	\(W_x\in\mathbb{R}^{d\times h}\)	\(\boldsymbol{y}_\ell\in\mathbb{R}^{v}\)
\(\boldsymbol{h}_\ell\in\mathbb{R}^{h}\)	\(W_h\in\mathbb{R}^{h\times h}\)	\(W_y\in\mathbb{R}^{h\times v}\)

\(f_\theta\): \(\theta=(W_x,W_h,\boldsymbol{b})\). \(g_\phi\): \(\phi=(W_y,\boldsymbol{c})\).

Unroll these shared functions to see the computation across the sequence.

About three minutes. Reuse ReLU already taught; vanilla RNNs often use tanh, but the state/parameter-sharing idea does not require a new activation today. b has h entries and c has v entries. x_W_x and h_{−1} W_h are both h-entry row vectors, so their addition is defined. ReLU applies elementwise. The vocabulary head reuses multiclass prediction; it predicts w_{+1}. Distinguish h, a scalar feature width, from bold h_, a state vector. [Sources] source:d2l-rnn.

Unrolling repeats computation with shared parameters

Three drawn cells are three uses of one learned transition.

About 2.5 minutes. A diagram unfolds the sequence, not parameter ownership. Each h_depends on its own current embedding and the previous state; the states generally differ. The embedding for a repeated token comes from the same row of E. Read outputs as predictions of the next token, not reconstructions of the current token. Longer inputs reuse the same parameter set, with more recurrent steps. The three visible tokens are a short illustrative segment; the numeric trace explicitly sets the state before the segment to zero. [Sources] source:d2l-rnn.

A three-token trace separates input, state and prediction

Synthetic parameters: \(W_x=I_2\), \(W_h=0.5I_2\), \(\boldsymbol{b}=\boldsymbol{0}\), \(\boldsymbol{h}_0=\boldsymbol{0}\).

Token	Embedding \(\boldsymbol{x}_\ell\)	Updated state \(\boldsymbol{h}_\ell=\operatorname{ReLU}(\boldsymbol{x}_\ell+0.5\boldsymbol{h}_{\ell-1})\)
please	\([1,0]\)	\([1,0]\)
send	\([0,1]\)	\([0.5,1]\)
the	\([1,1]\)	\([1.25,1.5]\)

A head column \([1,1]^{\mathsf T}\) gives the report logit \(2.75\) at the last step.

That logit is a score; softmax still compares it with all vocabulary logits.

3.5 minutes. Work the second row explicitly, then let students calculate the third during the explanation. This is not a separate voting activity. The embeddings match the earlier synthetic table exactly. All preactivation entries are nonnegative here; ReLU leaves their values unchanged. The head has six columns in vocabulary order: [0,0], [1,0], [0,1], [−1,0], [1,1], [0,−1], with c=0. Each listed pair is a column of W_y. Thus the last output row is [0,1.25,1.5,−1.25,2.75,−1.5]. These values are verified by the figure/check script, not trained results. At this step report receives probability about 0.623, not 2.75. A single logit does not determine a probability. Resetting the state to zero before “please” is a simplified segment example, distinct from the earlier BOS training illustration.

Training assigns credit through the unrolled computation

4.5 minutes. Advance and rewind with normal Reveal controls. First show all forward operations: token ID lookup in E, state updates f_theta, vocabulary heads g_phi, logits, cross-entropy with the next-token targets, and the mean loss. The target labels at the loss nodes are send, the and report. All forward values are saved before backward begins; no parameter update occurs during either traversal.

Then reverse the arrows: the mean sends 1/m to each loss; each local loss sends a partial derivative to its logits. Process the last state first, then the middle state, then the first. At each hidden state, sum contributions from its own vocabulary head and the next recurrent update before continuing backward. These are partial derivatives of the scalar loss, not new forward state values. Each repeated use contributes to the same theta/phi gradient. Backward through embedding lookup accumulates into the selected rows of E; discrete token IDs do not receive gradients. The initial zero state is fixed. Make one optimizer update after accumulating all shared-parameter gradients. The final frame is also the PDF summary. [Sources] source:d2l-bptt; source:d2l-rnn; source:pytorch-autograd-2-7.

Repeated derivatives can weaken or amplify distant credit

Scalar illustration: \(h_\ell=\operatorname{ReLU}(a h_{\ell-1}+x_\ell)\), with all relevant preactivations positive.

\[\frac{\partial h_\ell}{\partial h_{\ell-1}}=a, \qquad \frac{\partial h_\ell}{\partial h_{\ell-n}}= \prod_{j=\ell-n+1}^{\ell}\frac{\partial h_j}{\partial h_{j-1}}=a^n\]

Local derivative	10 steps	20 steps	100 steps
\(a=0.5\)	\(9.77\times10^{-4}\)	\(9.54\times10^{-7}\)	\(7.89\times10^{-31}\)
\(a=1.5\)	\(57.7\)	\(3.33\times10^3\)	\(4.07\times10^{17}\)

A longer path can make earlier information difficult to learn to preserve.

About 2.25 minutes. Use explicit partial derivatives throughout. The scalar example isolates one state-to-state path; it is not a formula for the entire parameter gradient. In vector RNNs local Jacobians multiply, and the loss also supplies other paths/contributions. ReLU derivatives can additionally be zero at negative preactivations; the active-path assumption makes the multiplication transparent and avoids its kink at zero. Do not claim that every long sequence necessarily fails or that state storage guarantees useful long-range credit. The remaining RNN section will introduce the retain/overwrite idea and motivate direct access through attention. [Sources] source:d2l-bptt.

A longer message reuses the same learned transition

An RNN was trained on 20-token messages and receives a 30-token message.

What must grow to process the longer message?

A. The number of learned transition matrices

B. The number of recurrent computation steps

C. The width of each hidden state

B. More steps reuse the same parameters and state width.

Three minutes including voting. Parameter sharing permits longer inputs; it does not guarantee generalization or reliable memory at those lengths. Next, improve the transition itself with gates. [Sources] source:d2l-rnn.

A GRU learns what to retain and what to replace

1 · Gates

\[\boldsymbol{u}_\ell=[\boldsymbol{h}_{\ell-1},\boldsymbol{x}_\ell]\] \[\color{#2D6F9F}{\boldsymbol{r}_\ell=\sigma(\boldsymbol{u}_\ell W_r),\quad \boldsymbol{z}_\ell=\sigma(\boldsymbol{u}_\ell W_z)}\]

2 · Candidate

\[\color{#A85A24}{\widetilde{\boldsymbol{h}}_\ell=\tanh\!\left( [\boldsymbol{r}_\ell\odot\boldsymbol{h}_{\ell-1},\boldsymbol{x}_\ell]W\right)}\]

3 · New recurrent state

\[\begin{aligned}\color{#2A7F78}{\boldsymbol{h}_\ell} &\color{#2A7F78}{=(1-\boldsymbol{z}_\ell)\odot\boldsymbol{h}_{\ell-1}}\\ &\color{#2A7F78}{\quad+\boldsymbol{z}_\ell\odot\widetilde{\boldsymbol{h}}_\ell} \end{aligned}\]

\(\boldsymbol{z}_\ell\approx\boldsymbol{0}\): retain old state. \(\boldsymbol{z}_\ell\approx\boldsymbol{1}\): use candidate.

LSTM: a common, more complex gated alternative with a separate cell state.

Five minutes. The same three numbered colors map equations to one connected GRU computational graph. Each input appears once; wires show all reuse, including both uses of the update gate. Dots mark branching connections; white gaps mark crossings without a connection. The complement block maps z to 1-z. Advance to focus gates, candidate, then state mixture; the last reveal restores the full diagram and the PDF shows this full state. The gate convention matches the instructor’s original slides: z admits candidate content. Biases omitted; row u has h+d entries and W_r,W_z,W are (h+d) by h. Sigmoid gives gates between zero and one; tanh gives candidates between minus one and one. Both gates depend on the previous state and current input. The gate z is distinct from the next slide’s layer-output z with a layer superscript. A direct retention path can help memory and gradients but does not eliminate recurrence or guarantee nonvanishing gradients; derivatives also pass through gates. This is not the exact reset-after/complemented-gate convention of nn.GRU. Name long short-term memory without deriving its gates. [Sources] source:olah-understanding-lstms-2015.

Deep RNNs stack recurrent layers

Horizontal: \(\boldsymbol{h}_\ell^{(k)}\), recurrent state. Vertical: \(\boldsymbol{z}_\ell^{(k)}\), layer representation.

For vanilla RNNs and GRUs, \(\boldsymbol{z}_\ell^{(k)}=\boldsymbol{h}_\ell^{(k)}\): same value, different role.

1.5-minute informational slide. Superscripts label layers, not powers; subscripts label sequence positions. Use h on horizontal recurrence arrows and z on vertical layer-output arrows. For standard vanilla RNNs/GRUs these values are equal, not two independently learned states. The top representation feeds a task head producing y. The GRU gate z on the preceding slide has no layer superscript and is explicitly a gate. Each row shares its parameters across positions; different rows have different parameters. A cell can be a vanilla RNN, GRU or LSTM. Depth and sequence length are distinct axes of composition. [Sources] source:d2l-rnn; source:vaswani-attention-2017.

Bidirectional RNNs combine left and right context

Useful when the whole input is available, such as tagging a complete sentence.

Not valid for next-token prediction if the backward state can see the target.

1.5 minutes. The forward and backward chains use separate parameters. Concatenate their states at corresponding positions; a later head uses the combined feature. Each direction still requires sequential computation. This is an architecture option, not part of our causal email predictor. [Sources] source:d2l-rnn; source:vaswani-attention-2017.

PyTorch runs the recurrence; backward() performs BPTT

ids and shifted targets: [B,L]. Each row is a separate email.

model = nn.ModuleDict({
    "embed": nn.Embedding(V, 64, padding_idx=PAD),
    "rnn": nn.RNN(64, 96, nonlinearity="relu", batch_first=True),
    "head": nn.Linear(96, V),
})
opt = torch.optim.Adam(model.parameters(), lr=0.002)

# Repeat for each training batch:
opt.zero_grad(set_to_none=True)
X = model["embed"](ids)                    # [B,L,64]
H, _ = model["rnn"](X)                     # [B,L,96]
Y = model["head"](H)                       # [B,L,V] logits
loss = F.cross_entropy(Y.transpose(1, 2), targets, ignore_index=PAD)
loss.backward()                           # BPTT: head → RNN → embeddings
nn.utils.clip_grad_norm_(model.parameters(), 1.0)
opt.step()                                # one shared-parameter update

Forward builds the graph. Backward accumulates gradients. The optimizer updates weights.

Three minutes. Imports are torch, torch.nn as nn, and torch.nn.functional as F. V is vocabulary size and PAD is the padding ID. Create the modules and optimizer once, not inside the batch loop; call model.train() during training. PyTorch nn.RNN unrolls the recurrence internally and autograd saves dependencies for backward. No manual backward loop and no detach between positions. Without an explicit initial state, every email starts from zero. Right padding has no effect on earlier valid states in this unidirectional model; ignore padded targets. The loss expects [B,V,L], hence transpose only for the loss interface. Do not apply softmax before cross_entropy. clip_grad_norm_ limits gradient magnitude after BPTT, not vanishing gradients or the sequential dependency. nn.RNN’s two bias vectors sum to the single bias in the mathematics. The experiment script packages these same layers in an EmailRNN class and uses this training step. An RNN model—not a GRU—is measured next. [Sources] source:pytorch-rnn-2-7; source:pytorch-autograd-2-7.

A small RNN improves held-out email next-token prediction

Measured AESLC email-body experiment: 13,415 train / 1,769 validation / 1,604 test emails.

Model	Test log loss ↓ (nats/token)	Exact next-token accuracy ↑
Bigram: use only the previous token	4.840	18.9%
One-layer RNN: use the prefix	4.454	23.3%

• At most 64 tokens/email; shared 4,000-token vocabulary; one training seed.
• 6.5% of test targets become <UNK>; accuracy on known targets is 17.3% vs. 21.8%.

Synthetic prompt, measured predictions: please send the → attached, following, latest.

Next comparison: freeze these data, tokens and metrics; replace the RNN with attention.

Data: Zhang & Tetreault, AESLC (2019), CC BY-NC-SA 4.0. Local run, October 26, 2026.

Three minutes. Actual local training and held-out predictions, not synthetic performance. The earlier hand-worked numbers remain synthetic. Reuse email bodies for next-token continuation; do not report this as the original subject-line-generation benchmark. A new deterministic split groups normalized subjects, exact normalized bodies and identical 32-token prefixes; exact bodies are deduplicated. These grouping proxies cannot verify true thread independence because thread IDs are absent. Quotes are stripped where recognized; numbers, email addresses and URLs are normalized. The vocabulary uses train prefixes only; unknown test words become UNK. Exact-token accuracy includes punctuation, EOS and UNK; known-target accuracy excludes UNK. There are 91,824 nonpadding test targets. Bigram probabilities use count smoothing toward the unigram distribution, with fixed pseudo-count 10. The RNN has 659,552 parameters (embedding 64, hidden 96, ReLU) and trains with Adam at .002 for eight epochs, selecting epoch seven by validation loss. Reset state for each email; no bidirectional access. Test perplexities are 126.53 for bigram and 85.97 for RNN; unigram log loss is 5.923. Do not call the reported gain statistically robust from one seed or infer long-context ability from a 64-token cap. The displayed synthetic prompt was fixed before test evaluation; it is a diagnostic, not a correctness score. Data, vocabulary, model configuration, hashes, training history and results are in lectures/experiments/email-completion-v1; reproduce with lectures/scripts/arc6_email_rnn.py. Later models must use this exact split/tokenization/context/scoring, select on validation, and report parameters/training budget separately. Lower token loss does not measure writer acceptance or usefulness. [Sources] source:aeslc-2019; source:pytorch-rnn-2-7.

Recurrent memory creates information and training bottlenecks

Limitation	Why it matters
Fixed-width memory	Earlier information must survive repeated state updates; a final-state summary must represent the whole sequence.
Long gradient paths	Distant credit passes through many Jacobians, complicating optimization.
Sequential computation	State \(\boldsymbol{h}_\ell\) must wait for \(\boldsymbol{h}_{\ell-1}\), limiting training parallelism across positions.

Next: retain all position representations and combine them directly.

Three minutes. RNNs can retain all their outputs, but their contextual computation still routes through the recurrent chain; do not claim all RNNs discard all earlier states. A single-vector encoder is the especially clear compression case. Gates help memory but do not remove the sequential dependency. Parallel computation during training was a major motivation for replacing recurrence with self-attention in transformers; RNNs remain useful in some settings. Distinguish optimization through long paths from hardware throughput. Attention shortens paths and removes this within-layer chain, not every training difficulty. [Sources] source:d2l-bptt; source:vaswani-attention-2017, sections 1 and 4.