Recurrent Neural Networks (RNN)

David I. Inouye

Dynamic Content · Last Updated: October 30, 2026

One transition combines the current token and the previous state

Vertical recurrent cell: token features enter from below, the state update feeds a vocabulary head above, and a loop carries state to the next sequence position.

State update

\[\boldsymbol{h}_\ell=f_\theta(\boldsymbol{x}_\ell,\boldsymbol{h}_{\ell-1})\]

Vocabulary head

\[\boldsymbol{y}_\ell=g_\phi(\boldsymbol{h}_\ell)\]

The state changes with the prefix. The parameters \(\theta,\phi\) are shared.

Next: choose a concrete \(f_\theta\) and \(g_\phi\).

Familiar layers implement the recurrent update

\[\boldsymbol{h}_\ell=\operatorname{ReLU}\!\left( W_x\boldsymbol{x}_{\ell}+W_h\boldsymbol{h}_{\ell-1}+\boldsymbol{b}\right)\]

\[\boldsymbol{y}_\ell=W_y\boldsymbol{h}_{\ell}+\boldsymbol{c}, \qquad\boldsymbol{p}_\ell=\operatorname{softmax}(\boldsymbol{y}_\ell)\]

Embedding and state	Learned matrices	Vocabulary output
\(\boldsymbol{x}_\ell\in\mathbb{R}^{d}\)	\(W_x\in\mathbb{R}^{h\times d}\)	\(\boldsymbol{y}_\ell\in\mathbb{R}^{v}\)
\(\boldsymbol{h}_\ell\in\mathbb{R}^{h}\)	\(W_h\in\mathbb{R}^{h\times h}\)	\(W_y\in\mathbb{R}^{v\times h}\)

\(f_\theta\): \(\theta=(W_x,W_h,\boldsymbol{b})\). \(g_\phi\): \(\phi=(W_y,\boldsymbol{c})\).

Unroll these shared functions to see the computation across the sequence.

Unrolling repeats computation with shared parameters

The rolled recurrent cell is expanded into three vertical input-update-head columns for please, send, the. State passes from the initial zero vector through three steps. Each step has the same transition parameters and the same vocabulary-head parameters.

Three drawn cells are three uses of one learned transition.

A three-token trace separates input, state and prediction

Synthetic parameters: \(W_x=I_2\), \(W_h=0.5I_2\), \(\boldsymbol{b}=\boldsymbol{0}\), \(\boldsymbol{h}_0=\boldsymbol{0}\).

Token	Embedding \(\boldsymbol{x}_\ell\)	Updated state \(\boldsymbol{h}_\ell=\operatorname{ReLU}(\boldsymbol{x}_\ell+0.5\boldsymbol{h}_{\ell-1})\)
`please`	\([1,0]^{\mathsf T}\)	\([1,0]^{\mathsf T}\)
`send`	\([0,1]^{\mathsf T}\)	\([0.5,1]^{\mathsf T}\)
`the`	\([1,1]^{\mathsf T}\)	\([1.25,1.5]^{\mathsf T}\)

A head row \([1,1]\) gives the report logit \(2.75\) at the last step.

That logit is a score; softmax still compares it with all vocabulary logits.

3.5 minutes. Work the second row explicitly, then let students calculate the third during the explanation. This is not a separate voting activity. The embeddings match the earlier synthetic table exactly. All preactivation entries are nonnegative here; ReLU leaves their values unchanged. The head has six rows in vocabulary order: [0,0], [1,0], [0,1], [−1,0], [1,1], [0,−1], with c=0. Each listed pair is a row of W_y. Thus the last output column is [0,1.25,1.5,−1.25,2.75,−1.5]^T. These values are verified by the figure/check script, not trained results. At this step report receives probability about 0.623, not 2.75. A single logit does not determine a probability. Resetting the state to zero before “please” is a simplified segment example, distinct from the earlier BOS training illustration.

Training assigns credit through the unrolled computation

Inputs and initial state

Look up embeddings

Compute first step

Compute second step

Compute third step

Average token losses

Backpropagate third step

Backpropagate second step and sum branches

Backpropagate first step

Complete forward and reverse computation graph

4.5 minutes. Advance and rewind with normal Reveal controls. First show all forward operations: token ID lookup in E, state updates f_theta, vocabulary heads g_phi, logits, cross-entropy with the next-token targets, and the mean loss. The target labels at the loss nodes are send, the and report. All forward values are saved before backward begins; no parameter update occurs during either traversal.

Then reverse the arrows: the mean sends 1/m to each loss; each local loss sends a partial derivative to its logits. Process the last state first, then the middle state, then the first. At each hidden state, sum contributions from its own vocabulary head and the next recurrent update before continuing backward. These are partial derivatives of the scalar loss, not new forward state values. Each repeated use contributes to the same theta/phi gradient. Backward through embedding lookup accumulates into the selected rows of E; discrete token IDs do not receive gradients. The initial zero state is fixed. Make one optimizer update after accumulating all shared-parameter gradients. The final frame is also the PDF summary. [Sources] source:d2l-bptt; source:d2l-rnn; source:pytorch-autograd-2-7.

Repeated derivatives can weaken or amplify distant credit

Scalar illustration: \(h_\ell=\operatorname{ReLU}(a h_{\ell-1}+x_\ell)\), with all relevant preactivations positive.

\[\frac{\partial h_\ell}{\partial h_{\ell-1}}=a, \qquad \frac{\partial h_\ell}{\partial h_{\ell-n}}= \prod_{j=\ell-n+1}^{\ell}\frac{\partial h_j}{\partial h_{j-1}}=a^n\]

Local derivative	10 steps	20 steps	100 steps
\(a=0.5\)	\(9.77\times10^{-4}\)	\(9.54\times10^{-7}\)	\(7.89\times10^{-31}\)
\(a=1.5\)	\(57.7\)	\(3.33\times10^3\)	\(4.07\times10^{17}\)

A longer path can make earlier information difficult to learn to preserve.

A longer message reuses the same learned transition

An RNN was trained on 20-token messages and receives a 30-token message.

What must grow to process the longer message?

A. The number of learned transition matrices

B. The number of recurrent computation steps

C. The width of each hidden state

B. More steps reuse the same parameters and state width.

A GRU learns what to retain and what to replace

1 · Gates

\[\boldsymbol{u}_\ell=[\boldsymbol{h}_{\ell-1}^{\mathsf T},\boldsymbol{x}_\ell^{\mathsf T}]^{\mathsf T}\] \[\color{#2D6F9F}{\boldsymbol{r}_\ell=\sigma(W_r\boldsymbol{u}_\ell),\quad \boldsymbol{z}_\ell=\sigma(W_z\boldsymbol{u}_\ell)}\]

2 · Candidate

\[\color{#A85A24}{\widetilde{\boldsymbol{h}}_\ell=\tanh\!\left( W\begin{bmatrix}\boldsymbol{r}_\ell\odot\boldsymbol{h}_{\ell-1}\\\boldsymbol{x}_\ell\end{bmatrix}\right)}\]

3 · New recurrent state

\[\begin{aligned}\color{#2A7F78}{\boldsymbol{h}_\ell} &\color{#2A7F78}{=(1-\boldsymbol{z}_\ell)\odot\boldsymbol{h}_{\ell-1}}\\ &\color{#2A7F78}{\quad+\boldsymbol{z}_\ell\odot\widetilde{\boldsymbol{h}}_\ell} \end{aligned}\]

One connected GRU graph: previous state and input feed both gates; reset gates the candidate path; update gate splits between the retained and candidate branches, which meet at the final sum.

Highlight gate computations from the concatenated previous state and current input.

Highlight reset multiplication, concatenation with input, and candidate transformation.

Highlight retained-state and candidate branches, weighted by one minus z and z and summed.

Complete connected GRU computational graph with all input, gate, candidate and output paths visible.

\(\boldsymbol{z}_\ell\approx\boldsymbol{0}\): retain old state. \(\boldsymbol{z}_\ell\approx\boldsymbol{1}\): use candidate.

LSTM: a common, more complex gated alternative with a separate cell state.

Five minutes. The same three numbered colors map equations to one connected GRU computational graph. Each input appears once; wires show all reuse, including both uses of the update gate. Dots mark branching connections; white gaps mark crossings without a connection. The complement block maps z to 1-z. Advance to focus gates, candidate, then state mixture; the last reveal restores the full diagram and the PDF shows this full state. The gate convention matches the instructor’s original slides: z admits candidate content. Biases omitted; column u has h+d entries and W_r,W_z,W are h by (h+d). Sigmoid gives gates between zero and one; tanh gives candidates between minus one and one. Both gates depend on the previous state and current input. The gate z is distinct from the next slide’s layer-output z with a layer superscript. A direct retention path can help memory and gradients but does not eliminate recurrence or guarantee nonvanishing gradients; derivatives also pass through gates. This is not the exact reset-after/complemented-gate convention of nn.GRU. Name long short-term memory without deriving its gates. [Sources] source:olah-understanding-lstms-2015.

Deep RNNs stack recurrent layers

Two recurrent layers across three positions: inputs feed layer one; z representations pass upward and h recurrent states pass horizontally, with explicit position subscripts and layer superscripts.

Horizontal: \(\boldsymbol{h}_\ell^{(k)}\), recurrent state. Vertical: \(\boldsymbol{z}_\ell^{(k)}\), layer representation.

For vanilla RNNs and GRUs, \(\boldsymbol{z}_\ell^{(k)}=\boldsymbol{h}_\ell^{(k)}\): same value, different role.

Bidirectional RNNs combine left and right context

One RNN traverses the input left to right and another right to left. Their states at each position are concatenated into one representation.

Useful when the whole input is available, such as tagging a complete sentence.

Not valid for next-token prediction if the backward state can see the target.

PyTorch runs the recurrence; `backward()` performs BPTT

ids and shifted targets: [B,L]. Each row is a separate email.

model = nn.ModuleDict({
    "embed": nn.Embedding(V, 64, padding_idx=PAD),
    "rnn": nn.RNN(64, 96, nonlinearity="relu", batch_first=True),
    "head": nn.Linear(96, V),
})
opt = torch.optim.Adam(model.parameters(), lr=0.002)

# Repeat for each training batch:
opt.zero_grad(set_to_none=True)
X = model["embed"](ids)                    # [B,L,64]
H, _ = model["rnn"](X)                     # [B,L,96]
Y = model["head"](H)                       # [B,L,V] logits
loss = F.cross_entropy(Y.transpose(1, 2), targets, ignore_index=PAD)
loss.backward()                           # BPTT: head → RNN → embeddings
nn.utils.clip_grad_norm_(model.parameters(), 1.0)
opt.step()                                # one shared-parameter update

Forward builds the graph. Backward accumulates gradients. The optimizer updates weights.

Three minutes. Imports are torch, torch.nn as nn, and torch.nn.functional as F. V is vocabulary size and PAD is the padding ID. Create the modules and optimizer once, not inside the batch loop; call model.train() during training. PyTorch nn.RNN unrolls the recurrence internally and autograd saves dependencies for backward. No manual backward loop and no detach between positions. Without an explicit initial state, every email starts from zero. Right padding has no effect on earlier valid states in this unidirectional model; ignore padded targets. The loss expects [B,V,L], hence transpose only for the loss interface. Do not apply softmax before cross_entropy. clip_grad_norm_ limits gradient magnitude after BPTT, not vanishing gradients or the sequential dependency. nn.RNN’s two bias vectors sum to the single bias in the mathematics. The experiment script packages these same layers in an EmailRNN class and uses this training step. An RNN model—not a GRU—is measured next. [Sources] source:pytorch-rnn-2-7; source:pytorch-autograd-2-7.

A small RNN improves held-out email next-token prediction

Measured AESLC email-body experiment: 13,415 train / 1,769 validation / 1,604 test emails.

Model	Test log loss ↓ (nats/token)	Exact next-token accuracy ↑
Bigram: use only the previous token	4.840	18.9%
One-layer RNN: use the prefix	4.454	23.3%

At most 64 tokens/email; shared 4,000-token vocabulary; one training seed.
6.5% of test targets become <UNK>; accuracy on known targets is 17.3% vs. 21.8%.

Synthetic prompt, measured predictions: please send the → attached, following, latest.

Next comparison: freeze these data, tokens and metrics; replace the RNN with attention.

Data: Zhang & Tetreault, AESLC (2019), CC BY-NC-SA 4.0. Local run, October 26, 2026.

Three minutes. Actual local training and held-out predictions, not synthetic performance. The earlier hand-worked numbers remain synthetic. Reuse email bodies for next-token continuation; do not report this as the original subject-line-generation benchmark. A new deterministic split groups normalized subjects, exact normalized bodies and identical 32-token prefixes; exact bodies are deduplicated. These grouping proxies cannot verify true thread independence because thread IDs are absent. Quotes are stripped where recognized; numbers, email addresses and URLs are normalized. The vocabulary uses train prefixes only; unknown test words become UNK. Exact-token accuracy includes punctuation, EOS and UNK; known-target accuracy excludes UNK. There are 91,824 nonpadding test targets. Bigram probabilities use count smoothing toward the unigram distribution, with fixed pseudo-count 10. The RNN has 659,552 parameters (embedding 64, hidden 96, ReLU) and trains with Adam at .002 for eight epochs, selecting epoch seven by validation loss. Reset state for each email; no bidirectional access. Test perplexities are 126.53 for bigram and 85.97 for RNN; unigram log loss is 5.923. Do not call the reported gain statistically robust from one seed or infer long-context ability from a 64-token cap. The displayed synthetic prompt was fixed before test evaluation; it is a diagnostic, not a correctness score. Data, vocabulary, model configuration, hashes, training history and results are in lectures/experiments/email-completion-v1; reproduce with lectures/scripts/arc6_email_rnn.py. Later models must use this exact split/tokenization/context/scoring, select on validation, and report parameters/training budget separately. Lower token loss does not measure writer acceptance or usefulness. [Sources] source:aeslc-2019; source:pytorch-rnn-2-7.

Recurrent memory creates information and training bottlenecks

Limitation	Why it matters
Fixed-width memory	Earlier information must survive repeated state updates; a final-state summary must represent the whole sequence.
Long gradient paths	Distant credit passes through many Jacobians, complicating optimization.
Sequential computation	State \(\boldsymbol{h}_\ell\) must wait for \(\boldsymbol{h}_{\ell-1}\), limiting training parallelism across positions.

Next: retain all position representations and combine them directly.