LSTM Explained: Forget Gate, Cell State, and Long-Term Memory in RNNs

A long short-term memory (LSTM) network is a recurrent architecture designed to keep information across many time steps. Vanilla RNNs theoretically can do the same, but in practice their gradients vanish (or explode) as sequences get longer. LSTMs add a cell state and gates so that useful signals can pass through time with less friction.

This LSTM tutorial covers the vanishing-gradient problem, the forget/input/output gates, how cell state differs from hidden state, GRU as a simpler cousin, and when transformers have largely replaced recurrence.

Why vanilla RNNs forget

A simple RNN updates a hidden state

$$\mathbf{h}_t=\tanh(W_{xh}\mathbf{x}_t+W_{hh}\mathbf{h}_{t-1}+\mathbf{b}).$$

Backpropagation through time multiplies Jacobians of $W_{hh}$ at every step. If those Jacobians have spectral radius $<1$, gradients shrink exponentially — the vanishing gradient problem. The network then trains recent tokens and ignores the start of the sequence. Exploding gradients are the opposite pathology and are usually clipped.

Language, audio, and sensor traces need memory of events that happened dozens or hundreds of steps ago. That is the job LSTMs were invented for (Hochreiter & Schmidhuber, 1997).

Cell state and three gates

An LSTM maintains two vectors at each time $t$:

• Cell state $\mathbf{c}_t$: the long-term conveyor belt.
• Hidden state $\mathbf{h}_t$: the short-term output exposed to the next layer and the next step.

Three sigmoid gates, each outputting values in $(0,1)$, control the conveyor:

Forget gate — what to drop from yesterday’s cell:

$$\mathbf{f}_t=\sigma(W_f[\mathbf{h}_{t-1},\mathbf{x}_t]+\mathbf{b}_f).$$

Input gate and candidate — what new information to write:

$$\mathbf{i}_t=\sigma(W_i[\mathbf{h}_{t-1},\mathbf{x}_t]+\mathbf{b}_i),\qquad \tilde{\mathbf{c}}_t=\tanh(W_c[\mathbf{h}_{t-1},\mathbf{x}_t]+\mathbf{b}_c).$$

Cell update:

$$\mathbf{c}_t=\mathbf{f}_t\odot\mathbf{c}_{t-1}+\mathbf{i}_t\odot\tilde{\mathbf{c}}_t.$$

Output gate — what to expose as $\mathbf{h}_t$:

$$\mathbf{o}_t=\sigma(W_o[\mathbf{h}_{t-1},\mathbf{x}_t]+\mathbf{b}_o),\qquad \mathbf{h}_t=\mathbf{o}_t\odot\tanh(\mathbf{c}_t).$$

Additive cell updates ($\mathbf{f}_t\odot\mathbf{c}_{t-1}+\cdots$) are the key trick: gradients can flow along $\mathbf{c}$ through time without being multiplied by a weight matrix at every step.

The LSTM Calculator shows forget, input, and output gates together with cell state — worth using while you read the equations.

LSTM versus GRU versus RNN

A GRU merges cell and hidden state and uses a reset/update pair. Empirically GRUs often match LSTMs; LSTMs sometimes win on very long dependencies. Neither is automatically best — validate.

Where LSTMs still matter

Transformers dominate large NLP, but LSTMs remain relevant when:

• Sequences are long and compute budgets forbid full self-attention $O(T^2)$.
• You need a streaming model that consumes one tick at a time (some IoT and speech pipelines).
• The dataset is small and a huge transformer overfits.
• You are learning the ideas of gated recurrence before attention.

Time-series forecasting, remaining-useful-life models, and some recommendation session models still ship LSTMs in production.

Training tips

• Clip gradients (e.g. global norm 1 or 5).
• Pack padded sequences so padding does not pollute the last hidden state.
• Use teacher forcing in seq2seq, but mix in scheduled sampling if exposure bias hurts.
• Bidirectional LSTMs help when the whole sequence is available (named entity recognition); they cannot be used for strict forecasting into the future.
• Stacking more than two LSTM layers often needs residual connections and dropout on non-recurrent connections.

Frequently asked questions

What does the forget gate forget?

It scales each component of the previous cell state toward zero. A value near 0 wipes that memory channel; a value near 1 keeps it. The network learns which channels to reset after a delimiter (for example, a period in text or a regime change in a sensor).

Is LSTM a type of neural network?

Yes. It is a recurrent neural network with a specific cell. You still stack layers, choose hidden size, and train with backpropagation through time.

LSTM vs transformer — which should I learn first?

Learn a tiny RNN, then LSTM gates, then self-attention. Attention is easier to parallelize; gates teach you how gradient highways are engineered.

Why is my LSTM worse than a moving average on stock prices?

Markets are close to noisy; a flexible sequence model will overfit last week’s wiggle. Use walk-forward validation, simple baselines, and features that are actually causal. Architecture cannot fix a non-predictable target.

Next steps

Write the four LSTM equations on paper and track $\mathbf{c}_t$ for a sequence of length 4 with a forget gate stuck at 1 versus 0. Then open the LSTM Calculator and the RNN Calculator side by side. When you need global context in one shot, continue with transformers and self-attention.