Transformers Explained: Self-Attention, BERT, GPT, and Positional Encoding

A transformer is a neural architecture that mixes information across a sequence using self-attention instead of recurrence. It is the backbone of BERT, GPT, T5, and most modern large language models. The famous 2017 paper titled the idea bluntly: Attention Is All You Need.

This guide explains queries, keys, and values, scaled dot-product attention, multi-head attention, positional encoding, and the encoder-versus-decoder split that distinguishes BERT from GPT.

The problem attention solves

An RNN (or LSTM) reads tokens left to right. Distant dependencies must survive a chain of hidden states. Attention lets every position look at every other position in one step. For sequence length $T$, self-attention is $O(T^2)$ in time and memory — expensive for very long context, but highly parallel on GPUs.

That parallelism, not just accuracy, is why transformers scaled.

Queries, keys, values

Each token embedding $\mathbf{x}_i$ is linearly mapped to three vectors:

$$\mathbf{q}_i=W_Q\mathbf{x}_i,\quad \mathbf{k}_i=W_K\mathbf{x}_i,\quad \mathbf{v}_i=W_V\mathbf{x}_i.$$

Query asks “what am I looking for?” Key answers “what do I contain?” Value is the content that will be mixed in if the match is strong.

Scaled dot-product attention for a stack of tokens is

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\Big(\frac{QK^\top}{\sqrt{d_k}}\Big)V.$$

The $\sqrt{d_k}$ scale keeps dot products from saturating the softmax when $d_k$ is large. The softmax row for token $i$ is a probability distribution over tokens $j$. The output is a weighted sum of value vectors.

In a sentence such as “The animal did not cross the street because it was tired,” attention can learn to send “it” toward “animal” rather than “street.”

Explore the pieces in the Transformer Calculator: self-attention, multi-head attention, and positional encoding are easier to remember when you can change head count and sequence length.

Multi-head attention

One set of $W_Q,W_K,W_V$ learns one type of relation. Multi-head attention runs $h$ attention operations in parallel with different projections, then concatenates:

$$\mathrm{MultiHead}(Q,K,V)=\mathrm{Concat}(\mathrm{head}_1,\ldots,\mathrm{head}_h)W_O.$$

Heads can specialize: syntax, coreference, positional patterns, rare-token copying. More heads are not automatically better; they cost memory and can become redundant.

Positional encoding

Self-attention is permutation-invariant: without extra information, it cannot tell “dog bites man” from “man bites dog.” Transformers add positional encodings — sinusoidal functions in the original paper, learned embeddings in many later models, relative positions or RoPE in modern LLMs.

Absolute sinusoidal encodings use

$$PE_{(pos,2j)}=\sin(pos/10000^{2j/d}),\qquad PE_{(pos,2j+1)}=\cos(pos/10000^{2j/d}).$$

They are added to token embeddings before the first layer.

Encoder, decoder, BERT, GPT

The original transformer has an encoder stack (bidirectional self-attention over the source) and a decoder stack (masked self-attention over the target plus cross-attention into the encoder).

• BERT is encoder-only. Every token may attend to every other token. Pretraining is masked language modeling. Good for classification and span prediction.
• GPT is decoder-only. Attention is causal: token $t$ may not look at $t+1,t+2,\ldots$. That matches next-token prediction.
• T5 / original transformer use encoder-decoder for translation and other sequence-to-sequence tasks.

The block around attention is standard: residual connection, layer normalization, and a two-layer MLP (feed-forward network) applied position-wise.

Transformers versus LSTMs

If you need the gated-memory story first, read LSTM explained. If you need the last-layer classifier math, see sigmoid and softmax.

Frequently asked questions

What is self-attention in one sentence?

Each token computes a weighted average of all tokens’ value vectors, with weights from how well its query matches their keys.

Why do we divide by $\sqrt{d_k}$?

Unscaled dot products grow with dimension, pushing softmax toward one-hot vectors and tiny gradients. Scaling stabilizes training.

Do I need a transformer to classify images?

Not necessarily. CNNs remain strong. Vision transformers (ViT) patch an image into tokens and apply the same attention stack. The CNN guide is the right starting point for spatial filters.

Why is GPT called generative?

Causal decoding produces the next token from previous tokens, repeatedly. That is generation. BERT’s bidirectional encoder is not usually used that way without extra heads.

Next steps

Compute a $3\times 3$ attention matrix by hand for tiny $d_k=2$ vectors. Confirm that rows of $\mathrm{softmax}(QK^\top/\sqrt{d_k})$ sum to 1. Then use the Transformer Calculator to vary heads and sequence length. For the optimization that trains these blocks, keep gradient descent and loss functions nearby.