RNN Limitations

Reading One Word at a Time

What if a model could read every word of a sentence at once instead of one by one?

Recurrent neural networks, or RNNs, do the opposite. They move along the symbol positions of a sequence, handling one word per step.

Each step produces a hidden state, a summary of everything read so far. The next step needs that summary before it can begin.

This chain is the root of both limits in this lesson.

The Sequential Bottleneck

Because step 3 needs the result of step 2, the steps cannot run side by side.

The source calls this inherently sequential: it precludes parallelization within training examples.

Notice how the top row forms a single waiting line, while the bottom row handles every word at the same moment.

Computers can do many calculations at the same time. An RNN leaves most of that power waiting in line.

A 100-word sentence means 100 steps in a row, no matter how fast each step is.

Why Long Sequences Hurt Most

The bottleneck gets worse as sequences get longer.

One usual trick for speed is batching: training on many examples together. But long sequences use lots of memory, so fewer examples fit in a batch.

So the model cannot speed up inside an example, and memory limits speeding up across examples.

Training becomes slow. Better engineering helps, but the fundamental constraint of sequential computation remains.

Distant Words, Long Paths

The second limit is about meaning. Consider: "The keys that I left on the kitchen table are missing."

To pick "are" instead of "is", the model must link it back to "keys".

In an RNN, information travels one step per word.

Path length grows linearly with distance: 1 word apart takes 1 step, 5 apart takes 5 steps, 20 apart takes 20 steps.

Longer paths make the connection harder to learn.

Attention Ignores Distance

Attention mechanisms offer a way out.

They let a model connect any two positions directly, modeling dependencies without regard to their distance in the input or output.

"Keys" and "are" become one hop apart, just like neighbors.

Attention has become a core part of strong sequence models. The Transformer builds on it to drop recurrence and process all words in parallel.

Spotting RNN Limits in Practice

Two limits, one root cause: computation that follows the sequence step by step.

Bottleneck: training slows as sequences get longer, and memory caps batching.

Long-range dependencies: related words far apart are harder to connect.

If your model trains slowly or struggles with long texts, ask whether recurrence is the cause. Attention-based designs like the Transformer target both problems.

A recurrent neural network computes the hidden states for all positions in a sentence _____

• one position at a time
• all at the same time

Answer: one position at a time

RNNs factor computation along symbol positions: each step needs the hidden state from the previous step, so positions are computed one after another, not all at once.

Two RNN limitations: the _____ bottleneck makes training slow because steps cannot run in _____, while _____ dependencies are hard to learn because distant words are many steps apart.

Answer: Two RNN limitations: the sequential bottleneck makes training slow because steps cannot run in parallel, while long-range dependencies are hard to learn because distant words are many steps apart.

The sequential bottleneck is about speed: step-by-step computation blocks parallel processing. The long-range dependency problem is about learning: the path between distant words grows with their distance.

Why can't an RNN parallelize computation across positions within a single training sentence?

• Each step needs the result of the previous step
• RNNs have too few parameters to split work
• Each word must be stored in a separate model
• Words in a sentence have no fixed order

Answer: Each step needs the result of the previous step

Each step depends on the previous step's hidden state, so step 5 cannot start until step 4 finishes. This dependency forces sequential computation regardless of hardware.