A Predictor Is Not An Assistant

What comes out of pretraining

After the first training phase, the model is extremely good at one thing: continuing text.

That is not the same as being useful.

Ask a raw pretrained model how do I fix a leaking tap? and a very reasonable continuation is:

How do I unblock a drain? How do I replace a washer? How much does a plumber cost?

It is not being unhelpful. On the internet, a question is very often followed by more questions. It is predicting correctly and answering nothing.

Teaching it to answer

The second stage fixes this by demonstration rather than instruction.

People write many examples of a request paired with a good response. The model is trained on those pairs.

What it learns is a pattern: in this kind of exchange, a question is followed by an answer to that question. Not a rule saying "answer questions". A learned regularity, in the same way everything else it knows is a learned regularity.

This is instruction tuning, and it is what converts a text predictor into something that behaves like an assistant.

Teaching it which answer

There is a third stage, because knowing to answer is not the same as knowing which answer is good.

People are shown alternative responses to the same request and pick the better one. Those preferences are used to push the model toward the kinds of responses people preferred.

This is where a great deal of the personality gets installed: the tone, the helpfulness, the willingness to explain, the refusals.

It is also where several habits get installed that nobody intended, because "what people picked" includes preferring agreement and preferring longer answers. Those are their own subject, and they come from here.

Same machine underneath

Worth being clear: none of these stages changed the mechanism.

It is still predicting one chunk at a time from a distribution. Instruction tuning and preference tuning changed what is likely, not how likelihood is turned into text.

Which is why a well-tuned assistant can still fabricate confidently. The tuning made it helpful and well-mannered. It did not install a fact-checker, because there is nothing in this process that could.