Safeguard
Supply Chain Security•

Transformer Architecture Explained: Attention Is All You Need

The transformer explained from Attention Is All You Need: encoder, decoder, attention worked by hand, training, results and what modern LLMs changed.

Safeguard Security Team
77 min read

The transformer is a neural network that reads a whole sequence at once and lets every token decide, through a weighted average called attention, which other tokens matter to it. It was introduced in June 2017 by eight researchers working at Google, in the paper "Attention Is All You Need" (Vaswani et al., NIPS 2017). It replaced the recurrent networks that had dominated machine translation, trained in a fraction of the time, and set new records on two translation benchmarks.

The large language models behind today's chat assistants and coding tools descend from that design. The modern versions whose designs are published, such as Llama and Mistral, drop half of it (the encoder), move the normalisation, change how positions are encoded and share some attention weights, but the core block is still the one in the paper: attention, then a small feed-forward network, each wrapped in a residual connection and a normalisation layer, stacked many times.

This post walks through the paper from start to finish. It covers the problem it solved, every component of the architecture with the exact sizes the authors used, the attention maths worked by hand on a three-token example, the training recipe, the results as the paper states them, and what later models changed. It is written for engineers, security people and technical leaders who want to understand the machine properly, not just the metaphors. You need school algebra and a rough idea of what a matrix multiply is. Every symbol is defined when it first appears, and every formula is written out in plain text.

It is the first post in a ten-part series on how LLMs work, built from the original papers. Each later post goes deeper on one part: tokenization and embeddings, attention and the KV cache, pretraining and scaling laws, fine-tuning and alignment, mixture of experts, inference, retrieval and tool use, long context, and finally security across the whole LLM stack.

Contents

  1. What problem did "Attention Is All You Need" solve?
  2. What does the Transformer architecture look like?
  3. How does data flow through the model, step by step?
  4. How does the Transformer turn tokens into vectors?
  5. How does sinusoidal positional encoding work?
  6. What is scaled dot-product attention?
  7. How does attention work on a real example?
  8. What is multi-head attention, and why eight heads?
  9. How do masked self-attention and cross attention work in the decoder?
  10. What does the feed-forward network do?
  11. Why does each sub-layer have a residual connection and layer norm?
  12. How does the model turn vectors back into words?
  13. How was the Transformer trained?
  14. How fast is self-attention compared with RNNs and convolutions?
  15. What results did the paper report?
  16. Who wrote "Attention Is All You Need", and what happened to them?
  17. What came next: BERT, GPT and decoder-only models?
  18. How do modern LLMs differ from the 2017 Transformer?
  19. What did the paper leave unsolved?
  20. How do you secure what you build on transformer models?
  21. The takeaway
  22. Frequently asked questions
  23. Sources

What problem did "Attention Is All You Need" solve?

In 2017 the best translation systems were recurrent neural networks that read a sentence one word at a time. That made them slow to train, because step 5 had to wait for step 4, and weak at linking distant words. The Transformer removed recurrence entirely and used attention alone, so every position could be computed in parallel.

To see why that mattered, it helps to know what the field looked like in the three years before the paper.

Vertical timeline of papers from LSTM in 1997 through sequence to sequence and Bahdanau attention in 2014, GNMT and ConvS2S, the Transformer in June 2017, GPT, BERT, GPT-2, T5, GPT-3, and refinements such as multi query attention, pre-LN, SwiGLU, RoPE, FlashAttention, LLaMA, grouped query attention and Mixtral

Figure 1. The papers this post covers, from the recurrent era to today. The Transformer sits in the middle, and almost everything after it is a refinement of its parts.

Recurrent networks and LSTMs

A recurrent neural network (RNN) processes a sequence the way you read a sentence aloud: one token at a time, left to right, carrying a running summary. At each step t it takes the current input token x_t and the previous summary h_(t-1), which is called the hidden state, and produces a new hidden state h_t.

h_t = f(h_(t-1), x_t)

x_t      the vector for the token at position t
h_(t-1)  the hidden state after reading tokens 1 to t-1
f        a learned function (matrix multiplies plus a squashing nonlinearity)

Plain RNNs forget quickly. When you train them with gradient descent, the error signal has to travel back through every step, and it shrinks or explodes as it goes. The long short-term memory cell (LSTM, Hochreiter and Schmidhuber, 1997) added gates: small learned switches that decide what to write into a memory cell, what to keep and what to read out. The gated recurrent unit (GRU), introduced by Cho et al. in 2014 and compared with LSTMs by Chung et al., is a simpler variant of the same idea. By 2016, stacked LSTMs were the standard tool for language.

Sequence to sequence and the fixed-length bottleneck

Translation maps one sequence to another of a different length. In 2014 two groups showed that you could do this with two RNNs: an encoder that reads the source sentence, and a decoder that writes the target sentence one word at a time. Sutskever, Vinyals and Le used deep LSTMs. Cho et al. used the GRU-style network they introduced in the same paper.

In this "sequence to sequence" design, the encoder's final hidden state was the only thing passed to the decoder. A 40-word sentence had to be squeezed into one fixed-size vector. That vector was a bottleneck, and quality fell off on long sentences.

Bahdanau attention (2014)

Bahdanau, Cho and Bengio (published at ICLR 2015) named the problem directly: they conjectured that the fixed-length vector was the bottleneck. Their fix was to keep all of the encoder's hidden states, one per source word, and let the decoder look back at them at every output step. For each word it was about to write, the decoder computed a score for every source position, turned the scores into weights that sum to 1, and took the weighted average of the encoder states. That weighted average is attention. When the decoder wrote the German "Katze", the weights would concentrate on the English "cat".

Bahdanau's score was computed by a small feed-forward network, which is why it is called additive attention. A year later, Luong, Pham and Manning tested simpler scoring functions, including a plain dot product between the decoder state and each encoder state. Remember that dot product. The Transformer is built on it.

Attention fixed the bottleneck, but it was bolted onto recurrent networks. The RNN was still there, still reading one token at a time.

Why recurrence was the bottleneck

The paper's introduction puts the core complaint in one line: recurrent models factor computation along the positions of the sequence, and this "inherently sequential nature precludes parallelization within training examples".

GPUs are fast because they do thousands of multiplications at the same time. An RNN cannot use that within a sentence, because h_5 depends on h_4, which depends on h_3. You can process many sentences side by side, but long sentences use a lot of memory, which limits how many fit in a batch. The strongest systems in the paper's own comparison table needed an estimated 10^19 to 10^21 floating point operations to train. Google's production system, GNMT (Wu et al., 2016), used 8 LSTM layers in the encoder and 8 in the decoder.

There was a second problem. For word 1 to influence word 40, its signal had to survive 39 updates of the hidden state. The paper calls this the path length between two positions, and argues that shorter paths make long-range dependencies easier to learn.

Two panels. Top: a recurrent encoder reads the cat sat on the mat one token at a time through hidden states h1 to h6, six sequential steps ending in a single vector c. Bottom: a self-attention layer connects every input token to every output in one parallel step

Figure 2. An RNN needs one step per token and a long path between distant words. A self-attention layer connects every pair of positions directly, in one parallel step.

Convolutions tried first: ByteNet and ConvS2S

The Transformer was not the first attempt to escape recurrence. ByteNet (Kalchbrenner et al., 2016) and ConvS2S (Gehring et al., May 2017, one month before the Transformer) used convolutional networks, which process all positions in parallel. Their weakness was reach: a convolution only sees a small window, so connecting two distant words needs many stacked layers. The paper notes that the number of operations needed to relate two positions grows linearly with distance for ConvS2S and logarithmically for ByteNet. In the Transformer it is constant.

Attention had also started to work without an outside encoder. Self-attention, sometimes called intra-attention, relates different positions of a single sequence to each other, and had been used for reading comprehension and other tasks, for example by Cheng, Dong and Lapata (2016). But in almost every case it still ran alongside a recurrent network.

The authors' bet was simple to state and bold at the time: drop recurrence and convolution entirely, and build the whole encoder and decoder from attention plus small per-position networks. The paper's own conclusion calls the result "the first sequence transduction model based entirely on attention".

What does the Transformer architecture look like?

The Transformer is an encoder and a decoder, each a stack of six identical layers. Encoder layers have self-attention and a feed-forward network. Decoder layers add masked self-attention and attention over the encoder output. Every sub-layer has a residual connection and layer norm, and every vector is 512 numbers wide.

The paper's Figure 1 is the picture most engineers have in mind when they hear the word transformer. Here it is redrawn, with the sizes from the paper written in.

The Transformer architecture redrawn: source tokens enter an input embedding and positional encoding, pass through an encoder layer repeated six times with multi-head self-attention and feed forward sub-layers each followed by add and layer norm; target tokens shifted right enter an output embedding and positional encoding, pass through a decoder layer repeated six times with masked self-attention, cross attention over the encoder output and feed forward sub-layers; a linear layer and softmax produce next token probabilities; a side panel lists base model hyperparameters

Figure 3. The Transformer, redrawn from Figure 1 of the paper. Read it from the bottom: tokens go in at the bottom of each column, next-token probabilities come out at the top right.

The encoder

The encoder turns the source sentence into a sequence of vectors, one per source token, where each vector has been informed by the whole sentence. It is a stack of N = 6 identical layers. Each layer has two sub-layers:

  1. Multi-head self-attention. Every position looks at every position in the same sentence and mixes in information from the ones that matter.
  2. A position-wise feed-forward network. A small two-layer network applied to each position separately.

Around each sub-layer sits a residual connection (the input is added back to the output) followed by layer normalisation. The paper writes the output of each sub-layer as:

output = LayerNorm(x + Sublayer(x))

For these additions to work, every sub-layer and the embeddings produce vectors of the same width, d_model = 512.

The decoder

The decoder writes the target sentence one token at a time. It is also a stack of N = 6 layers, but each decoder layer has three sub-layers:

  1. Masked multi-head self-attention over the target tokens produced so far. The mask stops a position from looking at later positions, which would be cheating during training.
  2. Multi-head encoder-decoder attention, usually called cross attention today. Queries come from the decoder, keys and values come from the encoder's final output. This is where the decoder reads the source sentence.
  3. A position-wise feed-forward network.

Each again has a residual connection and layer norm. The decoder is autoregressive: at each step it consumes the tokens it has already generated as extra input when generating the next.

The head

On top of the decoder, a linear layer maps each 512-wide vector to one score per vocabulary entry, and a softmax turns those scores into probabilities for the next token.

The sizes, as published

HyperparameterSymbolBase modelBig model
Encoder and decoder layersN6 each6 each
Model width (every vector)d_model5121024
Feed-forward inner widthd_ff20484096
Attention headsh816
Key and value width per headd_k = d_v6464
DropoutP_drop0.10.3
Label smoothingepsilon_ls0.10.1
Training steps100,000300,000
Parameters65 million213 million

These come from Table 3 of the paper. The big model's per-head width is 1024 / 16 = 64, the same as the base model. The English to French big model used dropout 0.1 rather than 0.3.

How does data flow through the model, step by step?

Token IDs become 512-wide vectors, get position information added, and pass through six encoder layers that keep the shape fixed at batch by length by 512. The decoder does the same for the target, reads the encoder output through cross attention, and a final linear layer turns each position into a score for every vocabulary word.

Tensor shapes are the fastest way to understand a network. A tensor is just a multi-dimensional array of numbers, and its shape lists the size along each axis. We will use:

B   batch size: how many sentence pairs are processed together
n   source length in tokens
m   target length in tokens
V   vocabulary size (about 37,000 for English to German)

Two columns of processing steps with tensor shapes. Encoder: token IDs B by n, embeddings B by n by 512, split into 8 heads B by 8 by n by 64, attention scores B by 8 by n by n, back to B by n by 512, feed forward expansion to B by n by 2048. Decoder: target IDs B by m through masked self-attention scores B by 8 by m by m, cross attention scores B by 8 by m by n, to logits B by m by V

Figure 4. The data flow with tensor shapes, using a worked example of 2 sentence pairs, 20 source tokens and 18 target tokens. The shaded rows are where tensors grow: the attention score matrices, the 2048-wide feed-forward layer and the vocabulary-sized output.

Three things stand out once you follow the shapes.

First, the width never changes between layers. Each encoder layer takes a [B, n, 512] tensor and returns a [B, n, 512] tensor. That is what lets you stack six of them, or ninety-six, and what makes the residual additions possible.

Second, the attention scores are n by n per head. For a 20-token sentence that is 400 numbers per head, which is nothing. For a 100,000-token document it is 10 billion per head per layer. That quadratic term is the reason long-context models need special tricks, the subject of part 9 of this series.

Third, the decoder sees the encoder output in every one of its six layers. The encoder runs once. Its final [B, n, 512] output is reused as the keys and values of the cross attention in all six decoder layers.

Where the 65 million parameters live

The paper reports 65 million parameters for the base model. You can get close with arithmetic, which is a good check that you understand the parts. A parameter is one learned number.

Embedding matrix, shared:  37,000 x 512                        = 18.9 M

One attention block:       4 matrices of 512 x 512, plus biases = 1.05 M
One feed-forward block:    512 x 2048 + 2048 x 512, plus biases = 2.10 M
One layer norm:            2 x 512 (scale and shift)            = 0.001 M

Encoder layer: 1 attention + 1 feed-forward + 2 norms           = 3.15 M
Decoder layer: 2 attention + 1 feed-forward + 3 norms           = 4.20 M

6 encoder layers  = 18.9 M
6 decoder layers  = 25.2 M
Total             = 18.9 + 18.9 + 25.2  = about 63 M

Our count lands at about 63 million against the paper's 65 million. The vocabulary is "about 37000" in the paper, and implementation details account for the rest. Two things are worth noticing. The feed-forward blocks hold about twice as many weights as the attention blocks. And because one embedding matrix is shared three ways (explained in the next section), the vocabulary costs 19 million parameters instead of 57 million.

How does the Transformer turn tokens into vectors?

Each token ID selects one row of a learned embedding matrix, a list of 512 numbers that the model adjusts during training. The paper shares that matrix between the encoder input, the decoder input and the output layer, and multiplies the embeddings by the square root of 512, about 22.6.

Neural networks work on numbers, so text is first cut into tokens and each token is given an integer ID. The paper used subword tokens: byte-pair encoding (Sennrich et al., 2015) with a shared English and German vocabulary of about 37,000 tokens, and a 32,000-entry word-piece vocabulary for English to French. Subwords let the model handle rare words by splitting them, so "Wahlkampfspenden" might become three or four pieces rather than an unknown-word marker. Part 2 of this series covers tokenization in depth.

An embedding is a learned lookup table. For a vocabulary of V tokens and a model width of 512, it is a matrix with V rows and 512 columns. Token ID 4,182 simply selects row 4,182. At the start of training the rows are random. Gradient descent gradually moves tokens that behave alike towards each other in this 512-dimensional space.

E          embedding matrix, shape [V, 512]
token_id   an integer from 0 to V-1
x          = E[token_id] * sqrt(512)        shape [512]

Two details in section 3.4 of the paper are easy to miss.

One matrix, three jobs. The paper shares the same weight matrix between the input embedding, the output embedding and the final linear layer that produces the vocabulary scores. This trick, called weight tying, follows Press and Wolf (2016). It works here because English and German share one byte-pair vocabulary. It saves parameters and links two related questions: "what does this token mean as input?" and "how likely is this token as output?"

The scale factor. In the embedding layers the weights are multiplied by sqrt(d_model), which is sqrt(512), about 22.6. The paper does not explain why. The usual reading is that the embedding values start out small, while the positional encodings added next range from minus 1 to plus 1, so scaling up the embeddings stops the position signal from drowning out the token identity. Treat that as an interpretation, not something the authors state.

How does sinusoidal positional encoding work?

Attention by itself ignores word order, so the paper adds a position signal to every embedding. Each of the 512 dimensions is a sine or cosine wave of a different frequency, from a wavelength of about 6 positions up to roughly 60,000, giving every position a unique, smooth fingerprint.

Here is a fact that surprises people. If you shuffle the input tokens of a self-attention layer, the outputs are shuffled in the same way and are otherwise identical. Attention compares every token with every other token, but nothing in the computation says which came first. "Dog bites man" and "man bites dog" would look the same.

The fix is to inject position information directly into the vectors. The paper adds a positional encoding, a 512-wide vector that depends only on the position, to each token's embedding at the bottom of both stacks:

PE(pos, 2i)   = sin( pos / 10000^(2i / d_model) )
PE(pos, 2i+1) = cos( pos / 10000^(2i / d_model) )

pos      position in the sequence: 0, 1, 2, ...
i        index of a sine and cosine pair: 0, 1, ..., 255 when d_model = 512
2i       an even dimension (gets the sine)
2i+1     the odd dimension next to it (gets the cosine)

Each pair of dimensions is a clock hand turning at its own speed. For dimensions 0 and 1 the divisor is 10000^0 = 1, so the angle is just pos, and the wave repeats every 2 x pi, about 6.3 positions. As i grows, the divisor grows geometrically and the wave slows down. The last pair (i = 255) has a divisor of about 9,650, so its wavelength is about 60,600 positions. The paper describes exactly this pattern: the wavelengths form a geometric progression from 2 x pi up towards 10000 x 2 x pi.

A worked example for the first four dimensions:

divisor for i = 0:  10000^(0/512) = 1
divisor for i = 1:  10000^(2/512) = 1.0366

pos = 1:  [ sin(1),  cos(1),  sin(1/1.0366), cos(1/1.0366) ]
       =  [ 0.8415,  0.5403,  0.8219,        0.5697       ]

pos = 2:  [ sin(2),  cos(2),  sin(2/1.0366), cos(2/1.0366) ]
       =  [ 0.9093, -0.4161,  0.9364,       -0.3509       ]

Top: line chart of positional encoding dimensions 0, 128 and 256 over positions 0 to 100, showing a fast wave, a medium wave and a slow rising curve. Bottom: heatmap of the encoding for positions 0 to 39 and dimensions 0 to 127, with fast alternating stripes on the left and smooth bands on the right

Figure 5. Sinusoidal positional encoding for d_model = 512. Low dimensions oscillate quickly and tell nearby positions apart. High dimensions change slowly and tell far-apart positions apart. Together, each row is a unique pattern.

Why sines and cosines?

The authors give two reasons.

Relative positions become easy to compute. They hypothesised that the model could learn to attend by relative position, because for any fixed offset k, PE(pos + k) is a linear function of PE(pos). You can see why with one sine and cosine pair. Moving forward k positions is the same as rotating the pair by a fixed angle, and a rotation is a matrix multiply:

Let w be the frequency of one pair: w = 1 / 10000^(2i/d_model)

[ sin(w(pos+k)) ]   [  cos(wk)  sin(wk) ]   [ sin(w pos) ]
[ cos(w(pos+k)) ] = [ -sin(wk)  cos(wk) ] x [ cos(w pos) ]

The 2 by 2 matrix depends only on the offset k, not on pos. So "three tokens to the left" looks the same wherever you are in the sentence. Hold on to this idea of rotating pairs of dimensions. Four years later it became RoPE, the position method used by LLaMA and many other modern LLMs.

Hoped-for extrapolation. The authors also tested learned positional embeddings, a separate trainable vector for each position, and found "nearly identical results" (Table 3, row E: 25.7 BLEU on the development set against 25.8 for sinusoids). They kept sinusoids because they "may allow the model to extrapolate to sequence lengths longer than the ones encountered during training."

That hope did not fully hold. In 2021, Press, Smith and Lewis showed that language models with sinusoidal position embeddings extrapolate poorly to sequences longer than they were trained on, and proposed ALiBi as an alternative. Part 2 of this series covers learned embeddings, RoPE and ALiBi; part 9 covers how today's models stretch to very long contexts.

What is scaled dot-product attention?

Attention turns each token into a query, a key and a value. A token's query is compared with every key by a dot product, the scores are divided by the square root of the key width and passed through a softmax to get weights that sum to 1, and the output is the weighted average of the values.

This is the heart of the paper, and the formula fits on one line:

Attention(Q, K, V) = softmax( Q x K^T / sqrt(d_k) ) x V

Before taking it apart, here is the intuition behind the three letters.

Queries, keys and values

Think of a library catalogue. You arrive with a question (the query). Every book has a label on its spine (the key). You compare your question with every label, decide how relevant each book is, and then read the books (the values) in proportion to their relevance.

In self-attention, every token plays all three roles at once. Each token's vector x is multiplied by three different learned matrices to produce its query, key and value:

q = x W_Q      "what am I looking for?"
k = x W_K      "what do I contain, for matching purposes?"
v = x W_V      "what will I hand over if someone attends to me?"

x            the token's 512-wide vector at this layer
W_Q, W_K     learned matrices, 512 by d_k
W_V          learned matrix, 512 by d_v

Separating keys from values matters. The features that make a token easy to find (its key) need not be the same as the information it should pass on (its value).

The five operations

Stack the queries of all n tokens as the rows of a matrix Q, and do the same for K and V. Then:

  1. Score. Multiply Q by the transpose of K. Entry (i, j) of the result is the dot product of query i with key j: multiply matching entries and add them up. A large positive dot product means the two vectors point the same way. The result is an n by n score matrix.
  2. Scale. Divide every score by sqrt(d_k). In the paper d_k = 64, so the divisor is 8.
  3. Mask (decoder only). Set the scores a position is not allowed to see to minus infinity.
  4. Softmax. Apply softmax to each row. Softmax exponentiates each number and divides by the row total, so every entry becomes positive and each row sums to 1. These are the attention weights.
  5. Mix. Multiply the weights by V. Row i of the output is the weighted average of all value vectors, using row i's weights.

Left: a pipeline from Q, K and V through matrix multiply, scale by square root of d_k, optional mask, row-wise softmax and a second matrix multiply with V. Right: bar chart comparing softmax weights for scores 0, 4, 1 without scaling (0.017, 0.936, 0.047) and with scaling (0.100, 0.736, 0.164)

Figure 6. Scaled dot-product attention as five matrix operations. On the right, the same toy scores with and without the scaling step: without it, softmax already puts 94 percent of the weight on one key at d_k = 4, and at d_k = 64 the effect is far stronger.

Why divide by the square root of d_k?

This is the "scaled" in the name, and the paper's footnote 4 explains it in two sentences. Suppose the entries of a query q and a key k are independent random numbers with mean 0 and variance 1. Their dot product is the sum of d_k products, each with mean 0 and variance 1, so the sum has mean 0 and variance d_k.

q . k = q_1 k_1 + q_2 k_2 + ... + q_dk k_dk
mean(q . k)     = 0
variance(q . k) = d_k          so the standard deviation is sqrt(d_k)

d_k = 64:   typical scores are around plus or minus 8
after dividing by sqrt(64) = 8:   typical scores are around plus or minus 1

Softmax is very sensitive to large inputs. With scores spread over plus or minus 8 or more, one score usually dominates, its weight approaches 1, and the rest approach 0. In that regime the softmax is "saturated": nudging the scores barely changes the output, so gradients become tiny and learning stalls. The authors suspected exactly this, writing that large dot products push the softmax "into regions where it has extremely small gradients". Dividing by sqrt(d_k) brings the variance back to 1.

Dot-product versus additive attention

The paper compares its choice with Bahdanau's additive attention. The two are similar in theoretical complexity, but dot-product attention "is much faster and more space-efficient in practice", because it is just matrix multiplication, which GPUs and their libraries are built for. Additive attention had been reported to beat unscaled dot-product attention for large d_k, and the scaling factor is the authors' fix for that.

How does attention work on a real example?

Take three tokens, "the cat sat", with four-number queries, keys and values. Multiply queries by keys to get a 3 by 3 score table, divide by 2, apply softmax to each row, then use each row of weights to average the value vectors. The arithmetic fits on one page.

Real models use 64-number heads and thousands of tokens, but the operations are identical at any size. Here they are by hand with d_k = 4, so the scale factor is sqrt(4) = 2. The numbers are invented to keep the arithmetic easy. In a real model they would come from multiplying each token's vector by the learned matrices W_Q, W_K and W_V.

           Q (queries)        K (keys)          V (values)
the     [0, 0, 2, 0]       [1, 0, 1, 0]      [1, 0, 2, 0]
cat     [0, 1, 0, 2]       [0, 2, 0, 1]      [0, 2, 0, 1]
sat     [1, 1, 1, 1]       [1, 1, 0, 0]      [2, 1, 0, 1]

Step 1: scores. Each entry is a query row dotted with a key row. For example, the score of "cat" against itself:

q_cat . k_cat = 0x0 + 1x2 + 0x0 + 2x1 = 4

Full score matrix S = Q x K^T (rows are queries, columns are keys):

            the   cat   sat
the      [   2,    0,    0 ]
cat      [   0,    4,    1 ]
sat      [   2,    3,    2 ]

Step 2: scale. Divide by sqrt(4) = 2:

            the   cat   sat
the      [   1,    0,    0 ]
cat      [   0,    2,  0.5 ]
sat      [   1,  1.5,    1 ]

Step 3: softmax each row. Take e to the power of each entry and divide by the row total. For the "cat" row:

e^0 = 1.000     e^2 = 7.389     e^0.5 = 1.649     total = 10.038

weights = [1.000 / 10.038,  7.389 / 10.038,  1.649 / 10.038]
        = [0.100,           0.736,           0.164]

All three rows:

            the     cat     sat
the      [ 0.576,  0.212,  0.212 ]
cat      [ 0.100,  0.736,  0.164 ]
sat      [ 0.274,  0.452,  0.274 ]

Step 4: weighted average of the values. Row i of the output is the weights of row i applied to the value vectors:

out_the = 0.576 x [1,0,2,0] + 0.212 x [0,2,0,1] + 0.212 x [2,1,0,1]
        = [1.00, 0.64, 1.15, 0.42]

out_cat = 0.100 x [1,0,2,0] + 0.736 x [0,2,0,1] + 0.164 x [2,1,0,1]
        = [0.43, 1.64, 0.20, 0.90]

out_sat = 0.274 x [1,0,2,0] + 0.452 x [0,2,0,1] + 0.274 x [2,1,0,1]
        = [0.82, 1.18, 0.55, 0.73]

Each output is a blend of the three value vectors, weighted by relevance. "cat" took 73.6 percent of its new representation from its own value and the rest from its neighbours. In a trained model, the patterns are meaningful: a verb's query might match the keys of its subject, or a pronoun's query might match the noun it refers to.

Worked attention example with three tokens: query, key and value matrices; the raw score matrix; the scores divided by 2; the softmax weights as an Iris heatmap with values 0.576, 0.212, 0.212 for the, 0.100, 0.736, 0.164 for cat and 0.274, 0.452, 0.274 for sat; the output matrix; and the same weights under a causal mask, where the gets 1.000 on itself and cat gets 0.119 and 0.881

Figure 7. The three-token example in full. The causal version at the bottom right is what a decoder computes: blocked cells become minus infinity before the softmax, so they receive zero weight.

The same computation in code

Here is scaled dot-product attention in about ten lines of NumPy, run on the numbers above. It reproduces the weights and outputs exactly.

import numpy as np

def softmax(x, axis=-1):
    x = x - x.max(axis=axis, keepdims=True)   # subtract the row max for numerical safety
    e = np.exp(x)
    return e / e.sum(axis=axis, keepdims=True)

def attention(Q, K, V, mask=None):
    d_k = Q.shape[-1]
    scores = Q @ K.swapaxes(-1, -2) / np.sqrt(d_k)    # [..., n_q, n_k]
    if mask is not None:
        scores = np.where(mask, scores, -1e9)          # blocked cells become a huge negative
    weights = softmax(scores, axis=-1)                 # each row sums to 1
    return weights @ V, weights                        # [..., n_q, d_v]

Q = np.array([[0, 0, 2, 0], [0, 1, 0, 2], [1, 1, 1, 1]], dtype=float)
K = np.array([[1, 0, 1, 0], [0, 2, 0, 1], [1, 1, 0, 0]], dtype=float)
V = np.array([[1, 0, 2, 0], [0, 2, 0, 1], [2, 1, 0, 1]], dtype=float)

out, w = attention(Q, K, V)
print(np.round(w, 3))     # [[0.576 0.212 0.212] [0.1 0.736 0.164] [0.274 0.452 0.274]]
print(np.round(out, 2))   # [[1. 0.64 1.15 0.42] [0.43 1.64 0.2 0.9] [0.82 1.18 0.55 0.73]]

causal = np.tril(np.ones((3, 3), dtype=bool))       # True where attention is allowed
_, w_masked = attention(Q, K, V, mask=causal)
print(np.round(w_masked, 3))   # [[1. 0. 0.] [0.119 0.881 0.] [0.274 0.452 0.274]]

Subtracting the row maximum before exponentiating does not change the softmax result, and it stops large scores from overflowing. Real implementations use a large negative number rather than true minus infinity for the same reason.

What is multi-head attention, and why eight heads?

Instead of one attention over 512-wide vectors, the paper runs eight attentions in parallel, each on its own 64-wide projection of the queries, keys and values. The eight 64-wide outputs are concatenated back to 512 and mixed by one more learned matrix. The total cost is about the same as one full-width head.

One attention head produces one set of weights per token: a single opinion about which other tokens matter. Language needs several opinions at once. When processing "it" in "the server rejected the request because it was malformed", one useful signal is the noun "it" refers to, and another is the verb "it" belongs to. A single softmax has to average those into one distribution. The paper puts it plainly: with a single head, "averaging inhibits" attending to different kinds of information at different positions.

Multi-head attention gives each head its own learned projections, so each head can specialise:

head_i       = Attention( Q W_i^Q,  K W_i^K,  V W_i^V )      for i = 1 to h
MultiHead    = Concat(head_1, ..., head_h) x W^O

W_i^Q, W_i^K   512 by d_k  =  512 by 64
W_i^V          512 by d_v  =  512 by 64
W^O            (h x d_v) by 512  =  512 by 512
h = 8,   d_k = d_v = d_model / h = 64

Multi-head attention diagram: a 512-wide input is projected by eight heads, each with its own 512 by 64 query, key and value matrices; each head runs softmax of Q K transposed over 8 times V to produce 64 numbers; the eight 64-wide outputs are concatenated to 512 and multiplied by W_O, a 512 by 512 matrix, to give the 512-wide output

Figure 8. Split, attend, concatenate, project. Each head works in its own 64-dimensional subspace. W_O then mixes what the heads found.

Why the cost stays the same

Because each head works on 64 dimensions instead of 512, eight heads cost roughly what one 512-wide head would. The parameter count is identical: eight sets of 512 by 64 matrices is the same as one 512 by 512 matrix. In practice, implementations do exactly that. They multiply by one 512 by 512 matrix and then reshape the result from [n, 512] to [8, n, 64], so the eight heads run as one batched operation.

def multi_head_attention(x_q, x_kv, W_q, W_k, W_v, W_o, h, mask=None):
    n_q, d_model = x_q.shape
    n_k = x_kv.shape[0]
    d_k = d_model // h
    def split(t, n):                                  # [n, d_model] to [h, n, d_k]
        return t.reshape(n, h, d_k).transpose(1, 0, 2)
    Q = split(x_q @ W_q, n_q)
    K = split(x_kv @ W_k, n_k)
    V = split(x_kv @ W_v, n_k)
    heads, _ = attention(Q, K, V, mask)               # [h, n_q, d_k]
    concat = heads.transpose(1, 0, 2).reshape(n_q, d_model)
    return concat @ W_o                               # [n_q, d_model]

For self-attention you pass the same tensor as x_q and x_kv. For cross attention you pass the decoder state as x_q and the encoder output as x_kv. That one function covers all three uses in the paper.

What the ablations showed

The authors tested the number of heads while keeping computation constant (Table 3, rows A, measured on the English to German development set, newstest2013):

Heads hd_k = d_vDev BLEU
151224.9
412825.5
8 (base)6425.8
163225.8
321625.4

A single head was 0.9 BLEU worse than the best setting, and quality also dropped with too many heads. Rows B showed that shrinking the key size alone (d_k = 16 gave 25.1, d_k = 32 gave 25.4) hurt quality, which the authors read as a sign that working out compatibility between tokens is hard and might benefit from something richer than a dot product.

The paper's appendix includes attention visualisations where individual heads appear to track long-distance dependencies and pronoun references. The authors say many heads "appear to exhibit behavior related to the syntactic and semantic structure of the sentences." Later work has been more cautious about reading meaning into attention maps (see, for example, Jain and Wallace, 2019, "Attention is not Explanation"), but heads remain a common unit of analysis.

Multi-head attention also became the target of some of the biggest efficiency changes since 2017. Multi-query attention (Shazeer, 2019) and grouped-query attention (Ainslie et al., 2023) share keys and values across heads to shrink memory during generation. Part 3 of this series covers them, along with FlashAttention and the KV cache.

How do masked self-attention and cross attention work in the decoder?

The decoder uses attention twice per layer. Masked self-attention lets each target position look only at earlier target positions, so the model cannot peek at the word it is meant to predict. Cross attention then takes queries from the decoder and keys and values from the encoder, so each target word can read the whole source sentence.

The paper uses the same multi-head attention function in three places. What changes is where Q, K and V come from, and which cells are masked.

Three panels of attention mask grids. Encoder self-attention: a full 3 by 3 grid for the, cat, sat. Decoder masked self-attention: a 4 by 4 grid for BOS, die, Katze, sass with the upper triangle blocked by minus infinity. Cross attention: a full 4 by 3 grid with decoder positions as queries and encoder outputs as keys

Figure 9. The three uses of attention, translating "the cat sat" into German. Only the decoder's self-attention is masked.

Encoder self-attention

Queries, keys and values all come from the previous encoder layer. Every source position can attend to every other source position, left and right. This is what lets the encoder build representations in which "bank" in "river bank" differs from "bank" in "bank loan".

Decoder masked self-attention

During training, the decoder is given the whole correct target sentence at once, shifted one position to the right. If the target is "die Katze sass", the decoder's input is "BOS die Katze sass", where BOS is a start-of-sentence marker, and at each position it must predict the next word:

decoder input (shifted right):   BOS    die    Katze   sass
prediction target:               die    Katze  sass    EOS

This setup is called teacher forcing, and it lets all target positions train in parallel. But it creates a problem: in an unmasked self-attention layer, the position predicting "Katze" could simply look at "Katze" one position to its right.

The mask prevents that. Before the softmax, every score where the key position is later than the query position is set to minus infinity. Since e to the power of minus infinity is 0, those positions get exactly zero weight. The paper describes this as preventing "leftward information flow" to preserve the auto-regressive property. Combined with the one-position shift, the prediction for position i can depend only on the known outputs at positions before i.

mask for 4 target positions (1 = allowed, 0 = set to minus infinity):

            BOS  die  Katze  sass
BOS      [   1,   0,    0,    0 ]
die      [   1,   1,    0,    0 ]
Katze    [   1,   1,    1,    0 ]
sass     [   1,   1,    1,    1 ]

This lower-triangular pattern is called a causal mask. You saw its effect in the worked example: with the mask, "cat" could no longer see "sat", so its weights renormalised from 0.100, 0.736 and 0.164 to 0.119, 0.881 and 0.

The causal mask is the part of this paper that today's chat LLMs depend on most directly. A GPT-style model is, in effect, this masked self-attention stack with the encoder and cross attention removed.

Encoder-decoder cross attention

In the middle sub-layer of each decoder layer, the queries come from the decoder (the output of its masked self-attention sub-layer) and the keys and values come from the encoder's final output. The paper notes this "mimics the typical encoder-decoder attention mechanisms" of earlier sequence-to-sequence models, meaning Bahdanau-style attention. The difference is that it now uses scaled dot products, eight heads, and happens in every one of the six decoder layers.

No mask is needed here. The source sentence is fully known before translation starts, so every target position may read every source position. The score matrix is m by n: target length by source length.

What happens at inference time

Teacher forcing only works when you know the answer. To translate a new sentence, the model runs the encoder once and then generates the output one token at a time:

encoder_out = encoder(source)
output = [BOS]
repeat:
    probs = decoder(output, encoder_out)     # probabilities for the next token
    next  = choose a token from probs[-1]
    output.append(next)
until next == EOS or len(output) > len(source) + 50

The paper did not simply take the most likely token at each step. It used beam search with a beam size of 4: keep the four best partial translations at each step, extend each, and keep the best four again. It also used a length penalty of alpha = 0.6, following GNMT, so that short translations were not unfairly favoured, and capped the output at the input length plus 50 tokens, stopping early when possible. These settings were chosen on the development set.

Notice that this loop recomputes the decoder over the whole output so far at every step. Modern inference engines avoid that by caching each layer's keys and values, the KV cache. Part 7 of this series covers caching, batching and quantization.

What does the feed-forward network do?

After attention mixes information between tokens, a small two-layer network processes each token on its own: it expands the 512-wide vector to 2048, applies ReLU, which zeroes negative values, and projects back to 512. The same weights are used at every position, and these blocks hold about two thirds of each encoder layer's parameters.

Attention moves information between positions. It does not do much computation on each position by itself, because its output is a weighted average of value vectors. The second sub-layer in every encoder and decoder layer fills that gap:

FFN(x) = max(0, x W1 + b1) W2 + b2

x       one position's 512-wide vector
W1, b1  weights 512 by 2048, bias of 2048
max(0, .)  ReLU: keep positive numbers, replace negatives with 0
W2, b2  weights 2048 by 512, bias of 512

"Position-wise" means the same network is applied to each token separately and identically. Within a layer, token 1 and token 20 go through exactly the same W1 and W2. Different layers have different weights. The paper offers another way to see it: two convolutions with a kernel size of 1.

Why expand to four times the width and come back? The wider hidden layer gives the network room to represent many features, many of which are switched off by ReLU for any given input. Interpretability research later suggested these layers act partly as stored knowledge: Geva et al. (2020) argued that feed-forward layers in transformer language models behave like key-value memories, where the first matrix detects input patterns and the second promotes likely output tokens. That paper also notes that feed-forward layers make up about two thirds of a transformer's parameters, which matches our count above: 2.1 million against 1.05 million for attention in each encoder layer.

The ablations show the feed-forward width matters. Shrinking d_ff to 1024 dropped development BLEU from 25.8 to 25.4. Growing it to 4096 raised it to 26.2, at the cost of 90 million parameters instead of 65 million.

Many modern LLMs replace this block with a gated version, such as SwiGLU from Shazeer (2020), which uses three weight matrices instead of two. In some designs, such as Mixtral, the single feed-forward block becomes several "experts" with a router choosing which to use per token. Part 6 of this series covers mixture of experts.

Why does each sub-layer have a residual connection and layer norm?

A residual connection adds a sub-layer's input to its output, so information and gradients can skip straight through a deep stack. Layer normalisation rescales each vector to mean 0 and variance 1. The paper normalised after the addition (post-LN). GPT-2, GPT-3, LLaMA and other later LLMs normalise before the sub-layer instead (pre-LN), which trains more stably.

Residual connections

A residual connection, from He et al. (2015) and their ResNet image models, is the simplest idea in the network. Instead of replacing x with Sublayer(x), you add them:

y = x + Sublayer(x)

The sub-layer only has to learn the change it wants to make, not reproduce everything that passes through. During training, the gradient has a direct path backwards through every "+", so deep stacks remain trainable. It also explains why every sub-layer outputs 512 numbers: you can only add vectors of the same size.

A useful mental model for modern LLMs is the residual stream: one 512-wide (or 12,288-wide) vector per token that flows from the bottom of the network to the top, with every attention and feed-forward block reading from it and adding its contribution back.

Layer normalisation

Layer normalisation, from Ba, Kiros and Hinton (2016), keeps the numbers in a predictable range. For each token's vector separately, it subtracts the mean of its 512 entries, divides by their standard deviation, then applies a learned scale and shift:

mean  = (x_1 + ... + x_512) / 512
var   = ((x_1 - mean)^2 + ... + (x_512 - mean)^2) / 512
LN(x) = gamma * (x - mean) / sqrt(var + eps) + beta

gamma, beta   learned vectors of 512 numbers (scale and shift)
eps           a tiny constant to avoid dividing by zero

Unlike batch normalisation, it does not depend on the other examples in the batch, which makes it a good fit for variable-length text.

Post-LN in the paper, pre-LN later

The paper places the norm after the residual addition: LayerNorm(x + Sublayer(x)). This is now called post-LN. It works, but it is fussy to train. A deep post-LN stack needs a careful learning-rate warmup, which the paper used (more on that in the training section).

Later work moved the norm inside the residual branch, so that it is applied to the sub-layer's input:

post-LN (2017 paper):   y = LayerNorm( x + Sublayer(x) )
pre-LN (later models):  y = x + Sublayer( LayerNorm(x) )

GPT-2 (Radford et al., 2019) states that layer normalisation "was moved to the input of each sub-block", with an extra norm added after the final block. Wang et al. (2019) found pre-norm helped train much deeper translation models. Xiong et al. (2020) explained why: in post-LN, the gradients of parameters near the output are large at initialisation, so a large learning rate destabilises training and warmup is needed. With pre-LN the gradients are well behaved, and they trained pre-LN transformers without the warmup stage.

Left: the feed forward network expanding a 512-wide vector to 2048, applying ReLU and projecting back to 512. Middle: post-LN, where x passes through the sub-layer, is added to the residual and then normalised. Right: pre-LN, where x is normalised, passed through the sub-layer and added to the untouched residual

Figure 10. Between the attention layers. Left, the position-wise feed-forward network. Middle and right, the two ways of wrapping a sub-layer. In pre-LN the residual path from bottom to top never passes through a norm.

Many modern models also swap LayerNorm for RMSNorm (Zhang and Sennrich, 2019), which skips subtracting the mean and only divides by the root mean square. LLaMA (Touvron et al., 2023) uses pre-normalisation with RMSNorm.

Here is a complete post-LN encoder layer, as in the paper, built from the functions above:

def layer_norm(x, gamma, beta, eps=1e-6):
    mean = x.mean(axis=-1, keepdims=True)
    var = x.var(axis=-1, keepdims=True)
    return gamma * (x - mean) / np.sqrt(var + eps) + beta

def feed_forward(x, W1, b1, W2, b2):
    return np.maximum(0, x @ W1 + b1) @ W2 + b2       # ReLU in the middle

def encoder_layer(x, p, h=8):
    # Post-LN, as in the 2017 paper: LayerNorm(x + Sublayer(x))
    a = multi_head_attention(x, x, p["Wq"], p["Wk"], p["Wv"], p["Wo"], h)
    x = layer_norm(x + a, p["g1"], p["b1n"])
    f = feed_forward(x, p["W1"], p["b1"], p["W2"], p["b2"])
    return layer_norm(x + f, p["g2"], p["b2n"])

This omits dropout and biases on the attention projections, but the structure is exactly the paper's. Stack six of these, put the embedding and positional encoding underneath, and you have the encoder.

How does the model turn vectors back into words?

The decoder's final 512-wide vector at each position is multiplied by the transposed embedding matrix, giving one score, called a logit, per vocabulary token. Softmax converts the logits into probabilities. Because the paper ties weights, the output layer scores each token by how well the vector matches that token's own embedding.

The last step is short. For each target position, the decoder stack outputs a 512-wide vector. A linear layer maps it to V numbers, one per vocabulary entry:

logits = h x W_out          h: [512],   W_out: [512, V],   logits: [V]
probs  = softmax(logits)    V probabilities that sum to 1

With weight tying, W_out is the transpose of the embedding matrix E. The score for token j is the dot product of the decoder's vector with token j's embedding. In other words, the model predicts the token whose embedding best matches what it has computed. For English to German, V is about 37,000, so this is a 512 by 37,000 matrix multiply per position, one of the largest single operations in the base model.

During training, the probability assigned to the correct next token feeds the loss (cross entropy: minus the log of that probability, averaged over positions). During inference, beam search uses these probabilities to extend its candidates.

The paper also averaged checkpoints. For the base models it used a single model obtained by averaging the weights of the last 5 checkpoints, written at 10-minute intervals. For the big models it averaged the last 20. Weight averaging smooths out noise from the final steps of training.

How was the Transformer trained?

The base model trained for 12 hours on one machine with eight NVIDIA P100 GPUs, about 100,000 steps of 0.4 seconds each; the big model took 3.5 days. Training used WMT 2014 data, batches of about 25,000 source and 25,000 target tokens, the Adam optimizer with a warmup learning-rate schedule, dropout of 0.1 and label smoothing of 0.1.

Data and batching

English to GermanEnglish to French
DatasetWMT 2014WMT 2014
Sizeabout 4.5 million sentence pairs36 million sentences
Tokenizationbyte-pair encoding, shared vocabulary of about 37,000word-piece vocabulary of 32,000
Test setnewstest2014newstest2014

Sentence pairs were grouped by approximate length, so that each batch wasted little space on padding. Each training batch held roughly 25,000 source tokens and 25,000 target tokens.

Hardware and schedule

All training ran on one machine with 8 NVIDIA P100 GPUs.

ModelStep timeStepsWall-clock time
Baseabout 0.4 seconds100,00012 hours
Big1.0 seconds300,0003.5 days

For context, the introduction notes the Transformer could reach a new state of the art in translation quality "after being trained for as little as twelve hours on eight P100 GPUs."

The optimizer and the warmup schedule

The paper used the Adam optimizer (Kingma and Ba, 2014) with beta1 = 0.9, beta2 = 0.98 and epsilon = 10^-9. Adam keeps a running average of each parameter's gradient (controlled by beta1) and of its squared gradient (beta2), and scales each update by both, so every parameter gets its own effective step size.

The learning rate was not fixed. It followed this formula, equation 3 in the paper:

lrate = d_model^(-0.5) x min( step^(-0.5),  step x warmup_steps^(-1.5) )

d_model       = 512 for the base model
warmup_steps  = 4000
step          = the current training step, starting at 1

The min() picks whichever term is smaller. For the first 4,000 steps the second term is smaller, so the learning rate rises in a straight line. After step 4,000 the first term is smaller, so the rate decays in proportion to one over the square root of the step number. The two terms meet exactly at step 4,000.

step        1:   0.0000002
step    1,000:   0.0001747
step    4,000:   0.0006988   (peak)
step   10,000:   0.0004419
step  100,000:   0.0001398   (end of base training)

Line chart of learning rate against training step. The base model with d_model 512 rises linearly to a peak of 0.000699 at step 4,000 and decays to 0.000140 at step 100,000. The big model with d_model 1024 peaks at 0.000494 and decays to 0.000057 at step 300,000

Figure 11. The paper's learning-rate schedule for the base model (d_model 512) and the big model (d_model 1024). Linear warmup for 4,000 steps, then inverse square root decay.

Why warm up? At the start of training the weights are random and Adam's running averages are poor estimates, so large steps can throw the model into a bad region. As discussed above, Xiong et al. later showed that post-LN transformers in particular have large gradients near the output at initialisation, which makes warmup close to essential for them. The d_model^(-0.5) factor makes wider models use smaller learning rates.

Here is the schedule in code:

def lrate(step, d_model=512, warmup=4000):
    step = max(step, 1)
    return d_model ** -0.5 * min(step ** -0.5, step * warmup ** -1.5)

Regularisation: dropout and label smoothing

The paper used three kinds of regularisation, techniques that stop a model from memorising its training data.

Residual dropout. Dropout (Srivastava et al., 2014) randomly sets a fraction of values to zero during training, so the network cannot rely on any single unit. The paper applies it to the output of each sub-layer before it is added to the residual and normalised, and to the sums of the embeddings and positional encodings at the bottom of both stacks. The rate was 0.1 for the base model and 0.3 for the big English to German model. Removing dropout entirely dropped development BLEU from 25.8 to 24.6.

Label smoothing. Normally the training target is "100 percent on the correct token, 0 percent on everything else". Label smoothing (Szegedy et al., 2015) softens this. With epsilon_ls = 0.1, the model is trained towards 90 percent on the correct token, with the remaining 10 percent spread evenly over the whole vocabulary:

V = 37,000,  epsilon_ls = 0.1

target for the correct token:   0.9 + 0.1 / 37,000  = about 0.900003
target for every other token:   0.1 / 37,000        = about 0.0000027

The paper notes that this "hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score." The ablation shows the trade clearly. Without label smoothing the development perplexity was better (4.67 against 4.92) but BLEU was worse (25.3 against 25.8).

Perplexity, briefly, measures how surprised a model is by the correct answer. It is the exponential of the average cross-entropy loss, and lower is better. A perplexity of 4.92 means the model is, on average, about as uncertain as if it were choosing uniformly among roughly five tokens.

How fast is self-attention compared with RNNs and convolutions?

Per layer, self-attention costs on the order of n squared times d operations, where n is the sequence length and d the model width, against n times d squared for a recurrent layer. For sentences shorter than the width, attention is cheaper, needs a constant number of sequential steps instead of n, and connects any two positions in one hop.

Section 4 of the paper, "Why Self-Attention", compares layer types on three measures: total computation per layer, how much of it can run in parallel (the minimum number of sequential operations), and the maximum path length between any two positions. Table 1 summarises it.

Left: the paper's Table 1 comparing self-attention, recurrent, convolutional and restricted self-attention layers by complexity per layer, sequential operations and maximum path length, with worked numbers for d of 512 at sequence lengths 50, 512, 4,096 and 16,384. Right: log scale chart of multiply adds per layer, with the self-attention and recurrent lines crossing at n equals 512

Figure 12. Table 1 of the paper, and what it means in numbers. Below 512 tokens, a self-attention layer needs fewer operations than a recurrent one; above it, more. Sequential steps and path length favour attention at every length.

Reading Table 1

Layer type                  Complexity per layer   Sequential ops   Max path length
Self-attention              O(n^2 . d)             O(1)             O(1)
Recurrent                   O(n . d^2)             O(n)             O(n)
Convolutional               O(k . n . d^2)         O(1)             O(log_k(n))
Self-attention (restricted) O(r . n . d)           O(1)             O(n/r)

n = sequence length, d = representation width, k = convolution kernel size,
r = size of the neighbourhood in restricted self-attention

The "O" notation describes how cost grows, ignoring constant factors.

Complexity per layer. Self-attention computes a score for every pair of positions, n squared pairs, each a d-wide dot product: n^2 times d. A recurrent layer does one d by d matrix multiply per position: n times d^2. Which is bigger depends on whether n or d is bigger. The paper points out that in 2017 machine translation, sentences represented with word pieces or byte pairs were usually shorter than d, so self-attention was cheaper. With d = 512 and a 50-token sentence, attention needs about 1.3 million multiply-adds per layer against 13.1 million for recurrence.

Sequential operations. This is the decisive column for GPUs. A recurrent layer needs n steps in a row. Self-attention needs a constant number, regardless of length.

Maximum path length. In a recurrent layer, information from the first token reaches the last after n steps. In self-attention it is one hop. Convolutions sit in between: a stack of dilated convolutions needs about log_k(n) layers to connect everything.

The paper also floated restricted self-attention, where each position only attends to a neighbourhood of size r, cutting cost to r times n times d at the price of a longer path, n divided by r. The authors said they planned to investigate it further. Later work on sparse and local attention, such as Sparse Transformers (Child et al., 2019) and Longformer (Beltagy et al., 2020), picked up that idea. Part 9 of this series covers long context.

The table also holds the Transformer's main weakness, stated honestly by the authors themselves: the n^2 term. Today's models handle contexts of hundreds of thousands of tokens, far beyond the crossover point, so the quadratic cost of attention is a real engineering problem. FlashAttention (Dao et al., 2022) computes exact attention in tiles to cut memory traffic, which is one of the main reasons long contexts became practical.

As a side benefit, the authors suggested self-attention "could yield more interpretable models", because you can inspect the attention weights directly.

What results did the paper report?

On WMT 2014 English to German, the big Transformer scored 28.4 BLEU, more than 2 points above the best earlier results including ensembles. On English to French it scored 41.8 BLEU, a new single-model best, after 3.5 days on eight GPUs. Even the base model beat every earlier English to German system at a fraction of the training cost.

What BLEU measures

BLEU (bilingual evaluation understudy) scores a machine translation by how many of its word sequences, of length one to four, also appear in human reference translations, with a penalty for translations that are too short. It runs from 0 to 100 and higher is better. On a mature benchmark like WMT 2014, where systems had been competing for years, gains of a point or two were treated as real progress.

Table 2: translation quality and training cost

Left: horizontal bar chart of English to German BLEU: ByteNet 23.75, GNMT plus RL 24.6, ConvS2S 25.16, MoE 26.03, GNMT plus RL ensemble 26.30, ConvS2S ensemble 26.36, Transformer base 27.3, Transformer big 28.4. Right: dot plot of training cost in FLOPs on a log scale, with Transformer base lowest at 3.3e18 and the GNMT plus RL ensemble highest at 1.8e20

Figure 13. English to German results from Table 2. Higher BLEU is better. Training cost is the authors' estimate in floating point operations; lower is better.

ModelEN-DE BLEUEN-FR BLEUEN-DE cost (FLOPs)EN-FR cost (FLOPs)
ByteNet23.75
Deep-Att + PosUnk39.21.0 x 10^20
GNMT + RL24.639.922.3 x 10^191.4 x 10^20
ConvS2S25.1640.469.6 x 10^181.5 x 10^20
MoE (Shazeer et al., 2017)26.0340.562.0 x 10^191.2 x 10^20
Deep-Att + PosUnk ensemble40.48.0 x 10^20
GNMT + RL ensemble26.3041.161.8 x 10^201.1 x 10^21
ConvS2S ensemble26.3641.297.7 x 10^191.2 x 10^21
Transformer (base)27.338.13.3 x 10^183.3 x 10^18
Transformer (big)28.441.82.3 x 10^192.3 x 10^19

The paper prints one training-cost figure per Transformer model, spanning both language pairs, so the same number appears in both cost columns here.

A few points worth drawing out:

  • English to German. The big model's 28.4 beat the best previous result, the ConvS2S ensemble at 26.36, by more than 2 BLEU. An ensemble combines several separately trained models, so beating ensembles with one model was notable. The base model, at 27.3, also beat every previous single model and ensemble, with an estimated training cost of 3.3 x 10^18 FLOPs, about a third of the cheapest competitor in the table (ConvS2S at 9.6 x 10^18).
  • English to French. The big model's 41.8 was a new single-model state of the art, at less than a quarter of the training cost of the previous best model. In Table 2 it is also above the best ensemble (ConvS2S at 41.29), although the paper's own text, which quotes 41.0, claims only the single-model record.
  • A discrepancy in the paper. The abstract and Table 2 give 41.8 for English to French, but the results text in section 6.1 says 41.0. The current arXiv version (v7) keeps both numbers. We quote 41.8 because it appears in the table and abstract, and flag the difference for anyone citing it.
  • How cost was estimated. The authors multiplied training time, the number of GPUs and an estimate of each GPU's sustained single-precision throughput (they used 9.5 TFLOPS for the P100).

Table 3: what each component contributes

The model variations in Table 3 are the closest thing the paper has to a parts list with prices. All are measured on the English to German development set (newstest2013), without checkpoint averaging. The base model scores 25.8 BLEU there.

Change from baseDev BLEUParameters
Base model (N 6, d_model 512, d_ff 2048, h 8)25.865M
1 head of width 51224.9
32 heads of width 1625.4
Key width d_k = 1625.158M
2 layers instead of 623.736M
4 layers25.350M
8 layers25.580M
d_model 25624.528M
d_model 102426.0168M
d_ff 102425.453M
d_ff 409626.290M
No dropout24.6
Dropout 0.225.5
No label smoothing25.3
Label smoothing 0.225.7
Learned positional embeddings25.7
Big model (d_model 1024, d_ff 4096, h 16, dropout 0.3, 300K steps)26.4213M

The authors' summary is the honest one: bigger models are better, dropout is very helpful against over-fitting, a single head is clearly worse, and learned versus sinusoidal positions makes almost no difference.

Beyond translation: parsing

To test whether the architecture generalised, the authors trained a 4-layer Transformer with d_model = 1024 on English constituency parsing, which turns a sentence into a tree of its grammatical structure. Trained only on the Wall Street Journal portion of the Penn Treebank, about 40,000 sentences, it reached 91.3 F1 on section 23. In a semi-supervised setting with about 17 million sentences, it reached 92.7. The authors report this beat all previously reported models except the Recurrent Neural Network Grammar (their Table 4 also lists a multi-task result of 93.0). The authors note that, unlike earlier RNN sequence-to-sequence models, the Transformer beat the BerkeleyParser even with only the 40,000 WSJ sentences.

Who wrote "Attention Is All You Need", and what happened to them?

The paper has eight authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin, from Google Brain, Google Research and the University of Toronto. All are marked as equal contributors in random order, and by 2024 all eight had left Google.

The paper was presented at the 31st Conference on Neural Information Processing Systems (NIPS 2017, now NeurIPS) in Long Beach, California. It was first posted to arXiv on 12 June 2017, and the latest revision, version 7, is dated 2 August 2023. The code was released as part of Google's tensor2tensor library.

AuthorAffiliation on the paper
Ashish VaswaniGoogle Brain
Noam ShazeerGoogle Brain
Niki ParmarGoogle Research
Jakob UszkoreitGoogle Research
Llion JonesGoogle Research
Aidan N. GomezUniversity of Toronto (work performed while at Google Brain)
Łukasz KaiserGoogle Brain
Illia PolosukhinWork performed while at Google Research

Every name carries an asterisk for "Equal contribution", and the footnote adds "Listing order is random." The same footnote, unusually, spells out who did what. In summary: Jakob Uszkoreit proposed replacing RNNs with self-attention and started the effort to evaluate the idea. Ashish Vaswani, with Illia Polosukhin, designed and implemented the first Transformer models. Noam Shazeer proposed scaled dot-product attention, multi-head attention and the parameter-free position representation. Niki Parmar designed, implemented, tuned and evaluated many model variants. Llion Jones experimented with variants, built the initial codebase, and worked on efficient inference and the visualisations. Łukasz Kaiser and Aidan Gomez designed parts of and implemented tensor2tensor, which replaced the earlier codebase and sped up the research.

Steven Levy's 2024 account in WIRED, based on interviews with all eight, adds context. The name "transformer" was chosen early. The title was a late addition: Llion Jones suggested it a couple of nights before the deadline, by analogy with the Beatles song "All You Need Is Love". Reviewer responses were mixed but positive, and the paper was accepted for a poster session.

According to that article, as of March 2024 all eight authors had left Google. Vaswani and Parmar co-founded Adept and then Essential AI. Shazeer co-founded Character AI. Gomez co-founded Cohere in Toronto. Uszkoreit co-founded the biotech company Inceptive. Jones co-founded Sakana AI in Tokyo. Polosukhin co-founded NEAR. Kaiser joined OpenAI.

What came next: BERT, GPT and decoder-only models?

Within two years the field split the Transformer in half. OpenAI's GPT kept only the decoder and trained it to predict the next word on unlabelled text. Google's BERT kept only the encoder and trained it to fill in masked words. GPT's decoder-only route, scaled up, became the template for today's LLMs.

The 2017 paper was about translation, a task with paired input and output. The next leap came from realising that the architecture could learn from plain, unlabelled text, of which there is vastly more.

GPT (June 2018): decoder only

Radford et al. (2018) at OpenAI used a "12-layer decoder-only transformer with masked self-attention heads", with 768-dimensional states, 12 attention heads and 3,072-dimensional feed-forward layers. They removed the encoder and the cross attention, leaving a stack of causal self-attention and feed-forward blocks. The model was pretrained on the BooksCorpus dataset to predict the next token over 512-token sequences, then fine-tuned for specific tasks. It used learned position embeddings and the GELU activation instead of ReLU.

The WIRED account reports that OpenAI's chief researcher at the time, Ilya Sutskever, suggested that Alec Radford work on the transformer idea soon after the paper appeared, and that this led to the first GPT models.

BERT (October 2018): encoder only

Devlin et al. (2018) at Google took the other half. BERT is a bidirectional encoder: every token can see every other token, as in the original encoder. Because a model that can see the whole sentence cannot be trained to predict the next word, BERT was trained to predict randomly masked words (masked language modelling), plus a next-sentence prediction task. It came in two sizes: BERT-base (12 layers, hidden size 768, 12 heads, 110 million parameters) and BERT-large (24 layers, 1024, 16 heads, 340 million), trained on BooksCorpus (800 million words) and English Wikipedia (2,500 million words). Encoder models like BERT became the default for classification, search and embeddings. Google began applying BERT to search in 2019, according to WIRED.

GPT-2, T5 and GPT-3: scale

GPT-2 (February 2019) scaled the decoder-only design to 1.5 billion parameters (the paper's table lists 1,542 million), moved layer norm to the input of each sub-block (pre-LN), grew the vocabulary to 50,257 and the context from 512 to 1,024 tokens. T5 (Raffel et al., 2019) kept the full encoder-decoder and cast every task, from translation to classification, as text in and text out. Kaplan et al. (2020) measured how language-model loss falls as a power law with more parameters, data and compute. And GPT-3 (Brown et al., 2020) scaled the decoder to 175 billion parameters: 96 layers, d_model = 12,288, 96 heads of width 128, and a 2,048-token context.

The same blocks spread beyond text. The Vision Transformer (Dosovitskiy et al., 2020) cut images into 16 by 16 patches and treated them as tokens, fulfilling the authors' stated plan to extend the Transformer to "images, audio and video".

Why decoder-only won for LLMs

There is no single paper that settles this, so take the following as a summary of common reasoning rather than a proven result. A decoder-only model has one stack, one objective (predict the next token) and one way to use it (keep generating). Every token of every document is a training example, so it soaks up unlabelled text efficiently. Prompting and in-context learning, which GPT-3 showcased, fall out of the same mechanism: the "input" is just the beginning of the text the model continues. Encoder-decoder models remain strong for tasks with a clear input and output, such as translation, and encoder-only models remain common for embeddings and classification.

Part 4 of this series covers GPT-1 to GPT-3, BERT, pretraining objectives and the scaling laws of Kaplan and Chinchilla in depth.

How do modern LLMs differ from the 2017 Transformer?

The skeleton is unchanged: embeddings, then a stack of attention and feed-forward blocks with residual connections and normalisation, then a vocabulary projection. What changed is the detail: decoder-only stacks, pre-normalisation with RMSNorm, rotary position embeddings (RoPE), SwiGLU feed-forward layers and grouped-query attention, at vastly larger scale.

Comparison table of the 2017 Transformer and a typical modern decoder-only LLM across nine rows: overall shape, training task, norm placement, norm type, positions, feed forward, attention heads, attention masks and example scale

Figure 14. What survived and what changed between 2017 and today's LLMs.

LLaMA (Touvron et al., February 2023) is a useful reference point because its paper lists its departures from the original Transformer explicitly. Each one maps to a component you have now seen.

Pre-normalisation with RMSNorm. LLaMA normalises the input of each sub-layer rather than the output, crediting GPT-3, and uses RMSNorm. This is the pre-LN change described above, and it is why modern training recipes are less fragile than the 2017 one.

SwiGLU feed-forward. LLaMA replaces ReLU with the SwiGLU activation from Shazeer (2020), credited to PaLM, with a hidden size of 2/3 x 4d instead of 4d. A gated unit multiplies two linear projections of the input, one passed through a smooth activation, so the network learns which features to let through:

original FFN:  FFN(x)    = max(0, x W1) W2
SwiGLU FFN:    FFN(x)    = ( Swish(x W1) * (x W3) ) W2

Swish(z) = z * sigmoid(z)        * means element-by-element multiply

The third matrix is why the hidden size is cut to two thirds: it keeps the parameter count roughly the same.

Rotary position embeddings. LLaMA removes absolute positional embeddings and applies RoPE (Su et al., 2021) at every layer, credited to GPT-Neo. RoPE takes the rotation idea hidden inside the sinusoidal encoding and applies it directly to the queries and keys, rotating each pair of dimensions by an angle proportional to the position. The dot product between a query at position p and a key at position q then depends on their offset, p minus q, which builds relative position into attention itself. Part 2 of this series works through RoPE and ALiBi with numbers.

Grouped-query attention. In the original design each of the 8 heads has its own keys and values. During generation those keys and values must be stored for every previous token, and that memory becomes the bottleneck. Multi-query attention (Shazeer, 2019) shares one key and value head across all query heads. Grouped-query attention (Ainslie et al., 2023) is the middle ground: groups of query heads share a key and value head. Llama 2 uses GQA in its 34B and 70B models.

Causal masking everywhere. The decoder-only stack uses the decoder's causal mask in every layer and has no cross attention. The "prompt" plays the role the encoder input used to.

Mixture of experts. Some models replace each feed-forward block with several expert blocks and a router. Mixtral 8x7B has 8 feed-forward experts per layer and routes each token to 2 of them, so each token has access to 47 billion parameters but uses about 13 billion. Part 6 covers this.

Training and alignment. The 2017 model learned from 4.5 million translated sentence pairs. Modern LLMs pretrain on very large text corpora, then go through instruction tuning and preference training such as RLHF or DPO. Part 5 covers alignment and fine-tuning, including LoRA and QLoRA.

Using models, not just building them. Much of what makes an LLM product useful sits outside the network: retrieval, tool calls and protocols such as MCP. Part 8 covers retrieval-augmented generation and tool use.

That so little of the 2017 design had to be thrown away says a lot about how good it was.

What did the paper leave unsolved?

Four things stand out. Attention cost grows with the square of the sequence length, which the paper acknowledged. Post-LN training needed a careful warmup that later work explained. The hope that sinusoids would extrapolate to longer sequences did not hold well. And generation stayed one token at a time.

Fairness to the paper means noting its limits as clearly as its strengths.

  • The quadratic cost. The authors were open about it and proposed restricted attention as future work. It became the central engineering problem for long-context models, driving sparse attention, FlashAttention, ring attention and the other techniques in part 9.
  • Training fragility. The warmup schedule worked, but it was a workaround for an instability nobody had explained yet. Xiong et al. traced it to the placement of layer norm in 2020.
  • Position extrapolation. The paper chose sinusoids partly because they "may allow the model to extrapolate". ALiBi's authors later reported that sinusoidal models extrapolate poorly beyond their training length, and RoPE-based models need their own tricks, such as position interpolation (Chen et al., 2023), to stretch.
  • Inference speed. The architecture parallelises training, but generation is still one token at a time, and each step must read the stored keys and values of every previous token. The authors listed "making generation less sequential" as a research goal. Speculative decoding (Leviathan et al., 2022) and the other techniques in part 7 are the modern answers.

None of this diminishes the result. It does show that "attention is all you need" was a claim about recurrence, not a claim that the design was finished.

How do you secure what you build on transformer models?

A transformer is a file of weights plus the code that loads and serves it, and both arrive through a software supply chain. Secure the model files you load, the AI libraries you depend on, the agents you run and the tools those agents can reach.

This series is mainly about how the models work. But every team that builds on them inherits risks that have nothing to do with attention maths, and part 10 covers them in depth. Four practical controls follow directly from the architecture you have just read about.

Scan model files before you load them. The trained Transformer is just its parameters, saved to disk. Some formats are pure data. Others are not: PyTorch checkpoints saved with Python's pickle format can run arbitrary code the moment they are loaded. Safeguard's model scanning covers files in pickle and PyTorch, safetensors, GGUF and ONNX formats, including models pulled from Hugging Face, Kaggle and ModelScope, so you can see what a checkpoint contains, and how risky its format is, before it reaches a GPU host. Background: safetensors vs pickle vs GGUF and how to inspect a pickle model file without running it.

Know which AI library vulnerabilities you actually reach. The code around the weights (tokenizers, model loaders, serving frameworks and the provider SDKs your application calls) is ordinary open source with ordinary vulnerabilities. Safeguard's software composition analysis covers the full transitive dependency tree and uses reachability analysis to show whether your code actually calls the vulnerable function, so a team can fix the reachable vulnerability in its serving path before a theoretical one in an unused module. Where an upstream fix has not shipped, Safeguard can open a fix pull request your CI tests, or supply a patched build through Gold. See the AI and ML platform guide to securing models, notebooks and AI SDKs.

Put runtime guardrails around agents. A decoder-only model generates whatever text is most likely given its context, and that context can include instructions injected by an attacker. When the model drives an agent that calls tools, that becomes an action, not just text. Safeguard Guard applies runtime guardrails that you define, and gives every AI agent its own identity, so you can see and limit what each agent can access. More in Guard runtime guardrails and agent identity in Safeguard.

Govern MCP servers and connectors. The Model Context Protocol connects models to tools and data. Each MCP server is a new piece of supply chain with access to real systems. Safeguard's MCP Guardrail issues guardrail proxy URLs that sit in front of MCP servers and enforce policy on agent tool traffic, and Gold Open Source lets you check an MCP server before you connect it. See how MCP Guardrail works and what an MCP server is and how to secure it.

The takeaway

The Transformer's central idea is that a sequence model does not need to read in order. Give every token a query, a key and a value, let every token compare its query with every key, and average the values by relevance. Add position information so order is not lost, wrap each step in a residual connection and a normalisation layer, add a small per-token network, and stack the block. Because every step is a matrix multiply, the whole thing trains in parallel on GPUs, and that is what made scale possible.

Nearly a decade later, the details have moved: one stack instead of two, the norm before the sub-layer, rotations instead of added sinusoids, gated feed-forward layers, shared keys and values. The block from section 3 of the paper is still the unit that published LLMs repeat dozens of times. If you understand the three-token example in this post, you understand the operation a modern LLM performs in every layer and every head, for every token it writes.

The next post in the series takes the very first step of that pipeline apart: how text becomes tokens, and how tokens and positions become vectors.

Try it yourself

Click through these interactive Safeguard demos to see the steps described in this guide.

Test yourself

What problem did the Transformer solve?

The best 2017 translation systems were recurrent networks that read one word at a time, which made them slow to train and weak at linking distant words. The Transformer dropped recurrence and used attention alone, so every position could be computed in parallel.

Why does the model add a positional encoding?

Attention by itself ignores word order. The paper adds a sine and cosine signal of different frequencies to every embedding, giving each position a unique, smooth fingerprint.

What are queries, keys and values doing in scaled dot-product attention?

A token's query is compared with every key by a dot product. The scores are scaled, passed through a softmax to get weights that sum to 1, and the output is the weighted average of the values.

Why are the scores divided by the square root of d_k?

A dot product of d_k terms has variance d_k, so scores can get large. That pushes the softmax into a saturated region with tiny gradients, and dividing by the square root of d_k brings the variance back to 1.

What does multi-head attention do, and what does it cost?

It runs eight attentions in parallel, each on its own 64-wide projection, then concatenates the outputs back to 512 and mixes them with one learned matrix. The total cost is about the same as one full-width head.

How does the decoder stop itself from peeking at the answer?

Masked self-attention lets each target position look only at earlier target positions. Cross attention then lets each target word read the whole source sentence through the encoder output.

Frequently asked questions

What is the transformer architecture in simple terms?

A transformer is a neural network that processes all tokens of a sequence at once. In each layer, every token uses attention to take a weighted average of information from the other tokens, then passes the result through a small feed-forward network. Stacking many such layers produces the representations that power translation models and modern LLMs.

Why is the paper called "Attention Is All You Need"?

Because the model uses attention as its only mechanism for relating positions, with no recurrence and no convolution. Earlier translation systems combined attention with recurrent networks. According to WIRED's 2024 account, co-author Llion Jones proposed the title shortly before the deadline, as a play on the Beatles song "All You Need Is Love".

What are queries, keys and values in attention?

They are three vectors computed from each token by separate learned matrices. The query describes what a token is looking for, the key describes what it offers for matching, and the value is the information it passes on. Attention scores each query against every key, turns the scores into weights with a softmax, and averages the values with those weights.

Why do transformers need positional encoding?

Self-attention treats its input as a set: shuffle the tokens and the outputs are shuffled the same way, with nothing else changing. Without extra information the model cannot tell "dog bites man" from "man bites dog". The original paper adds sine and cosine waves of different frequencies to each embedding. Many modern LLMs, including the LLaMA family, use rotary position embeddings (RoPE) instead.

What is the difference between encoder-only, decoder-only and encoder-decoder transformers?

Encoder-only models such as BERT let every token see the whole input and suit classification and embeddings. Decoder-only models such as GPT and most chat LLMs use a causal mask so each token sees only earlier tokens, which suits text generation. Encoder-decoder models such as the original Transformer and T5 read an input with an encoder and generate an output with a decoder.

Why is attention divided by the square root of d_k?

If query and key entries have variance 1, their dot product has variance d_k, so with d_k = 64 typical scores are around plus or minus 8. Scores that large push the softmax towards one-hot outputs, where gradients become tiny. Dividing by sqrt(d_k) brings the variance back to 1 and keeps training stable.

Is GPT the same architecture as the original transformer?

GPT uses the decoder half of the original Transformer, without cross attention, trained to predict the next token. Later GPT-style models also moved layer norm before each sub-layer and changed other details. Modern open models such as LLaMA add RMSNorm, SwiGLU, rotary position embeddings and, in larger versions, grouped-query attention.

Why is self-attention quadratic in sequence length, and does it matter?

Self-attention scores every pair of tokens, so n tokens produce n squared scores per head per layer. For 2017's short sentences that was cheaper than recurrence. For contexts of hundreds of thousands of tokens it dominates memory and compute, which is why techniques such as FlashAttention, sparse attention and KV-cache optimisation exist.

Sources

Last reviewed: 16 November 2026.

Never miss an update

Weekly insights on software supply chain security, delivered to your inbox.

We email you a link to confirm, and only then add you to the list. Unsubscribe in one click. Privacy

Self-healing security runs on Safeguard.

Your first fix PR is minutes away.

No sales call required, even your agent can complete the purchase over MCP.