Words Become Directions

A warning before you start

Nothing else in this library requires this block.

You can use AI extremely well without knowing any of it. None of the practical decisions change: you still brief properly, ground in real sources, and verify proportionally.

This is here because "it predicts the next word" is an unsatisfying place to stop if you are the kind of person who wants to see the machinery. That is a good enough reason. Open it out of curiosity, and close it whenever the curiosity runs out.

There is no text inside

First thing to give up: there is no text in there.

The moment your sentence enters, each token is converted into a long list of numbers. Hundreds or thousands of them. That list is called a vector, and you can think of it as a position in a space with that many dimensions.

We cannot picture a 4,000-dimensional space, and we do not need to. Everything that matters works the same way in three dimensions.

Position carries meaning

These positions are not arbitrary. They are learned during training, and they arrange themselves so that related things end up near each other.

Words for animals cluster. Words for cities cluster somewhere else. Formal words separate from casual ones.

More striking: relationships become directions. The step from a country to its capital turns out to be roughly the same movement, in the same direction, whichever country you start from. The step from singular to plural is another consistent direction.

Nobody programmed that. It emerged, because arranging things that way is useful for predicting text.

Why this is the crucial trick

This is what makes everything else possible.

Once meaning is position, then similarity is distance, and relationships are arithmetic. A machine that can only multiply and add numbers can now operate on meaning, because meaning has been converted into the only thing such a machine can handle.

That conversion is the foundational move. Everything from here is manipulating those positions.

One problem remains

A single vector per token cannot be the whole story.

The word bank has one starting position, but it means different things in river bank and savings bank. A fixed position cannot represent both.

Something has to adjust each position based on the words around it.

That something is attention, and it is the next lesson.