RNN Limitations

Reading One Word at a Time

Imagine reading a sentence through a keyhole that shows one word at a time.

Notice that only one word sits inside the opening while every later word waits its turn behind it.

That is how a recurrent model works. It is a type of machine that reads a sequence step by step, passing a small memory forward from each position to the next.

Position two cannot be handled until position one finishes.

This step-by-step habit is the source of almost every problem in this lesson.

Why Sequential Means Slow

Computers are fast because they can do many small jobs at the same time. That is called parallel work.

Recurrent models block this. Since each step needs the result of the step before it, the machine has to wait.

Training a model means feeding it enormous amounts of text. Waiting at every single position adds up to very long training times.

Long Sequences Make It Worse

Short sentences hide the problem. Long ones expose it.

A 10-word input needs 10 waiting steps. A 500-word document needs 500.

The usual escape route is batching: processing several examples side by side. But long sequences eat up memory, so fewer examples fit into a batch at once.

So the trick that normally rescues speed stops working exactly when you need it most.

Distant Words Lose Touch

The second limitation is about meaning, not speed.

Related words are often far apart. In "The keys that I left on the kitchen table this morning are missing", keys and are must agree.

In a recurrent model, information has to travel through every position in between. The distance between related words grows step by step with how far apart they sit.

The longer the path, the harder that connection is to learn.

Two Problems, One Cause

Both weaknesses trace back to the same design choice.

Because computation is arranged along the positions of the sequence, work cannot be spread out, and information cannot take a shortcut.

Speed and long-range understanding suffer together.

Attention Removes the Distance

Attention is a mechanism that lets a model link two positions directly.

With attention, dependencies can be modelled without regard to their distance in the input or output sequence. The word keys can reach are in one move.

Attention has become an integral part of strong sequence models for exactly this reason.

Why This Sets Up Transformers

Keep two ideas from this lesson.

First, sequential computation blocks parallel work and slows training, especially on long inputs where memory limits batching.

Second, distance between related words grows with position, weakening long-range learning.

A design built on attention alone can process positions simultaneously and connect any two words directly. That design is the Transformer.

Can a recurrent model handle position two before position one has finished?

• Yes
• No

Answer: No

No. Each position needs the result and memory from the previous position, so the model must wait.

For recurrent models, _____ causes waiting between positions, while _____ make distant word connections harder to learn.

Answer: For recurrent models, sequential computation causes waiting between positions, while long-range dependencies make distant word connections harder to learn.

Sequential computation creates the speed problem because positions cannot be handled together. Long-range dependencies create the meaning problem because distant information must travel through many positions.

In the sentence “The keys that I left on the table are missing,” why is the relationship between “keys” and “are” difficult for an RNN?

• The words must be connected through intervening positions
• The two words always have opposite meanings
• The sentence contains too few words to process
• The model can directly connect only adjacent words

Answer: The words must be connected through intervening positions

An RNN must carry information through every position between “keys” and “are.” The longer this path, the harder the connection is to learn.