Era 2 · Foundations · 1997

11 LSTM

Long Short-Term Memory · Hochreiter & Schmidhuber · Neural Computation

🟧 read selectively~2–3 horiginal ↗

The gist in 20 seconds. An RNN that solves the vanishing gradient. The cell holds a state with an additive path through time (the "constant error carousel"), and three gates decide what to forget, what to write and what to emit. It learns dependencies hundreds of steps long — the workhorse of sequences until Transformers.

Context

An ordinary RNN keeps context in a hidden state, but under backprop through many steps the gradient decays exponentially (or explodes) → long-range dependencies are never learned. Hochreiter diagnosed this in his 1991 thesis; the LSTM is the cure.

The idea and the mechanism

The cell holds a state c_t with an additive path through time. The flow is controlled by three sigmoid gates (values 0..1):

f_t, i_t, o_t = σ(·) c̃_t = tanh(·)

c_t = f_t ⊙ c_t−1 + i_t ⊙ c̃_t h_t = o_t ⊙ tanh(c_t)

forget decides how much of the old state to erase; input — how much of the new to write; output — how much of the state to let out. The network learns for itself when to remember, when to forget, when to emit.

calculus Why the gradient does not vanish: the additive path

The key is the derivative of the state with respect to the previous state. From c_t = f_t ⊙ c_t−1 + …:

∂c_t∂c_t−1 = f_t (the direct path — the dominant term)

The gradient across T steps is a product:

∂c_T∂c₀ = ∏_t=1^T f_t

The gate f_t is learned: the network can set f_t ≈ 1 and preserve the gradient across hundreds of steps (that is the "constant error carousel"). Compare with an ordinary RNN, where ∂h_t/∂h_t−1 = W^⊤ diag(σ′): repeatedly multiplying such matrices either contracts (eigenvalues < 1 → decay) or explodes (> 1). An additive structure plus a gate — that is the whole cure. Strictly speaking, ∂c_t/∂c_t−1 = f_t is the direct path; the full Jacobian also contains a term through the gates, which themselves depend on h_t−1 (and hence on c_t−1). But it is the direct additive path that dominates — and that is what rescues the gradient.

NumPy One step of an LSTM cell

import numpy as np
sig = lambda z: 1 / (1 + np.exp(-z))

def lstm_cell(x, h, c, W, b):
    z = W @ np.concatenate([x, h]) + b   # one matvec for all the gates
    i, f, g, o = np.split(z, 4)
    i, f, o = sig(i), sig(f), sig(o)
    g = np.tanh(g)
    c = f * c + i * g       # forget the old + write the new
    h = o * np.tanh(c)      # what to let out
    return h, c

The state rides along a "conveyor": the forget gate (×f) wipes what is no longer needed, the input gate (+i·g) writes new material, the output gate (o) decides what to emit. The additive path preserves the gradient.

Analogy. A conveyor belt with three valves. "Memory" rides along the belt. The forget valve dumps what is no longer wanted off the belt, the input valve loads new material on, the output valve scoops off whatever is needed right now. The belt itself barely slows the load down — which is why information (and the gradient) arrives from far away.

Why it matters

Until Transformers, the LSTM was the main workhorse for sequences: translation, speech, handwriting, time series. Seq2seq (#21) and the first attention (#22) are built on LSTMs. And the idea that "an additive path rescues the gradient" will echo in the residual connections of ResNet (#27) and the Transformer.

Connections

→ leads to21. Seq2Seq

Seq2seq puts two LSTMs back to back (an encoder and a decoder) and gets end-to-end machine translation. The LSTM is the brick out of which the whole early "sequence → sequence" architecture is assembled.

↔ close relative27. ResNet

One and the same trick in two guises: an additive "shortcut" (c_t = c_t−1 + … in an LSTM; y = x + F(x) in ResNet) stops the gradient vanishing across many steps or layers.

↔ superseded by32. Transformer

An LSTM links distant elements sequentially — one step at a time. A Transformer links any two positions directly and in parallel through self-attention, removing both the speed problem and the difficulty of very long dependencies.

Questions worth asking

If the gates decide everything, why not set forget = 1 always — let it remember everything?

Then the state is never cleared and fills up with stale context that gets in the way of the new. The strength of the LSTM is exactly the learned balance: remember long in some places (f≈1), dump fast in others (f≈0). Incidentally, the original LSTM had no forget gate — it was added later (Gers, 2000), and without it the networks did indeed suffer from "not forgetting".

How does a GRU differ from an LSTM, and why is it sometimes better?

A GRU merges the state and the output into one vector and uses two gates instead of three — simpler, fewer parameters, faster. On many tasks the quality is comparable; on some the LSTM is slightly better thanks to its separate memory. The practical choice is usually settled empirically, and the difference is small.

If the LSTM solved long-range dependencies, why was the Transformer needed?

The LSTM solved the vanishing gradient, not the sequential nature of the computation: step t waits for step t−1, so training does not parallelize and is slow on long inputs. And "hundreds of steps" is still not "tens of thousands of tokens". The Transformer removes recurrence altogether — hence both the speed and the scale of context.

What to read in the original

Read the essentials — the design of the cell and the gates, plus the constant-error-carousel argument; the paper's dated experiments can be skipped.