Email completion: RNN and transformer evidence

Email Completion: RNN and Transformer Evidence

David I. Inouye

Dynamic Content · Last Updated: November 2, 2026

On short email prefixes, both models improve on the bigram

Return to email completion: same grouped split, 4,000-token vocabulary and 64-token cap.

Model	Test loss ↓ (nats/token)	Exact next-token accuracy ↑
Bigram	4.840	18.9%
Simple RNN	4.454	23.3%
Small causal transformer	4.445	22.8%

For “please send the”, both rank “attached” first; this is a diagnostic, not a quality score.

One seed, eight epochs each; no decisive architecture winner from this experiment.

Three minutes. These are measured results, not synthetic values. Frozen AESLC body-completion protocol: 13,415 train, 1,769 validation and 1,604 test emails, grouped by normalized subjects/duplicate prefixes as proxies rather than verified thread IDs. Test scores cover 91,824 targets; 6.5% are UNK. Known-target accuracy is 21.830% RNN and 21.824% transformer, reinforcing that aggregate accuracy alone can obscure what changed. The RNN is one ReLU recurrent layer with embedding 64 and hidden 96 (659,552 parameters). The transformer has two pre-norm causal layers, width 64, four heads, FF width 256, learned absolute positions, no dropout and an untied vocabulary head (620,192 parameters). Both use FP32 CPU Adam, learning rate .002, batch 64, gradient clipping 1, and eight predeclared epochs; lowest validation NLL selects epoch 7 RNN and epoch 8 transformer. Equal epochs are not equal FLOPs or equal tuning effort. Test used only after selection. This is a single descriptive comparison, not a significance result, controlled architecture ablation or claim about long context/large modern models. Exact evidence: lectures/experiments/email-completion-v1/results.json and lectures/experiments/email-transformer-v1/results.json. [Sources] source:aeslc-2019; source:pytorch-rnn-2-7; source:vaswani-attention-2017.

The smaller parameter count did not give the smaller training-memory peak

Measured CPU probe: FP32 Adam, batch 64, length 64, vocabulary 4,000; five training steps.

Quantity	Simple RNN	Small transformer
Parameters	659,552	620,192
FP32 parameter storage	2.52 MiB	2.37 MiB
Peak process memory	548 MiB	630 MiB

Process memory includes activations, optimizer state, runtime and allocator overhead—not just weights.

This is CPU process memory, not GPU VRAM or a universal architecture ranking.

Three minutes. Each model ran in a fresh process with identical random integer input/target shapes for five full forward/backward/Adam updates. This isolates a shape-matched execution probe from differences in training history; it is not measured on the entire corpus and does not use learned checkpoints. On macOS, resource.getrusage(RUSAGE_SELF).ru_maxrss reports peak resident process bytes. Recorded values are 547.855 MiB RNN and 629.914 MiB transformer; show rounded values. Both include the Python/PyTorch process, weights, gradients, Adam moments, activations, workspaces and allocator behavior. The output logits alone have 64644000 FP32 entries, or 62.5 MiB, and further loss/backward allocations contribute. Do not assign the entire difference to attention without profiling allocations. No GPU benchmark, throughput advantage, multiple-seed uncertainty or asymptotic memory measurement is claimed. One measured configuration each. This complements the systems explanation: recurrent decode can retain a fixed-size hidden state while transformer K/V grows with context; that decode-state comparison is not what this training-memory table measures. Evidence and measurement code: email-transformer-v1/results.json and scripts/arc6_email_transformer.py. [Sources] source:aeslc-2019; source:pytorch-rnn-2-7; source:vaswani-attention-2017.