Era 3 · The deep learning explosion · 2014

22 Attention (Bahdanau)

Neural Machine Translation by Jointly Learning to Align and Translate · Bahdanau, Cho & Bengio · ICLR 2015

🟥 read in full~45–60 minoriginal ↗

The gist in 20 seconds. Fixes the seq2seq bottleneck: instead of one vector, at every step the decoder decides for itself which source words to look at, via a weighted sum of the encoder states. Gave interpretable alignments and a jump in quality on long sentences. The conceptual seed of the Transformer.

Context

Seq2seq (#21) loses information on long inputs — everything is squeezed into a single vector. Bahdanau, Cho and Bengio lift that limit by inventing the attention mechanism.

The idea and the mechanism

Instead of one vector, at EVERY step the decoder computes attention weights over all encoder hidden states (how relevant each one is to the current decoder state), normalizes them with a softmax and takes the weighted sum — a "context vector" tailored to the current generation step. That way the model dynamically focuses on the source words it needs.

linear algebra Attention as weighted retrieval by relevance

Let h_1..n be the encoder states and s_t−1 the current decoder state. First a relevance score for each h_i (additive, through a small network):

score_i = v^⊤ tanh(W₁ h_i + W₂ s_t−1)

Then a softmax turns the scores into weights (summing to 1), and the context is their weighted sum:

α_i = e^score_iΣ_j e^score_j, c_t = Σ_i α_i h_i

This is "soft content-based addressing": the query (s) is matched against the keys (h), and out comes a weighted blend of the values. The weights α are interpretable on top of that — they give the alignment between the words of the translation and the source. Replace the additive score with a dot product q·k and you get exactly the attention of the Transformer.

PyTorch Bahdanau additive attention

import torch, torch.nn.functional as F

def attention(s, H, W1, W2, v):
    # s: decoder state; H: (n, d) encoder states
    scores = (v * torch.tanh(H @ W1 + s @ W2)).sum(-1)  # relevance of each hᵢ
    alpha  = F.softmax(scores, dim=0)                    # weights, Σ = 1
    context = (alpha.unsqueeze(-1) * H).sum(0)           # Σ αᵢ hᵢ
    return context, alpha

At each step the decoder weights all encoder states (thickness = weight α) and takes their sum as the context. Attention moves dynamically across the source.

Analogy. That same simultaneous interpreter has finally been allowed to keep the source text in front of him and run a finger along the word he needs while he translates. No need to memorize the whole sentence — at every step he looks exactly at what he is translating right now. The attention weights are precisely "where the finger is pointing".

Why it matters

It improved translation straight away (long sentences especially), gave interpretability (word alignment) and — above all — became the conceptual seed of the Transformer (#32): there the idea was pushed into self-attention and made the only mechanism connecting tokens.

Connections

← fixes21. Seq2Seq

A direct cure for the bottleneck: instead of compressing the whole input into one vector, the decoder keeps access to all the encoder states. Attention is the "add-on" that removed the main limitation of encoder-decoder.

→ leads to32. Transformer

The Transformer takes this idea and pushes it to the limit: it drops the recurrence, replaces the additive score with a scaled dot product and makes self-attention the only mechanism. All modern attention grew out of this.

↔ echoes6. Hopfield network

Attention is retrieval from an associative memory by similarity. "Modern Hopfield networks" show formally that softmax attention = one-step pattern recovery. The old "memory as a minimum" idea and the new "attention" turn out to be the same thing.

Questions worth asking

Attention weights get read as an "explanation" of the decision — how far can you trust that?

With care. For alignment in translation the α really are meaningful. But in general "attention ≠ explanation": there are papers showing you can change the attention weights a great deal while barely changing the output — so they are not unambiguously the "cause" of the decision. A useful visualization, but not proof of what the model "really" leans on.

How does Bahdanau's additive attention differ from the dot-product attention in the Transformer?

Bahdanau computes the score with a small network, v^⊤tanh(W₁h + W₂s) — flexible, but it needs trainable matrices and is slower. The Transformer just takes q·k/√d — no extra parameters, and it vectorizes perfectly on a GPU (one matrix multiply for the whole sequence). At scale, simplicity and parallelism matter more, which is why the dot product won.

If attention solves long-range dependencies, why keep the recurrence here at all?

In 2014 attention was an addition to the recurrent encoder-decoder, not a replacement: the states h_i are still produced by an RNN. The bold move — "let's drop recurrence entirely and keep only attention" — comes three years later, in the Transformer. Here attention is still riding on top of an LSTM.

What to read in the original

Worth reading in full — this is the source of attention, and without it modern LLMs do not make sense. The key parts are the attention formula and the reading of it as "soft retrieval by relevance".