What Churns And What Holds

The exhausting version

The way most people try to keep up is to follow releases. New model, read the announcement. New feature, watch the demo. New leaderboard, check the ranking.

This is exhausting and produces surprisingly little. You end up knowing which model is nominally best this month, which changes next month, and being no better at using any of them.

The alternative is to sort what you could learn into two piles, and only maintain one of them.

The perishable pile

Do not memorise:

• Model names and versions. Renamed and superseded constantly
• Which is best. Changes every few months, and per task anyway
• Prices and limits. Fall and rise
• Context window sizes. Grown by orders of magnitude
• Product names and menu locations. Reorganised regularly
• Specific capability claims. "It cannot do X" ages badly

All of this is lookup material. Find it when you need it, do not carry it.

The durable pile

These have held across several generations of models and show no sign of changing:

• It predicts rather than looks up, so fluent and correct are unrelated
• Context is working memory, not storage
• It will guess confidently rather than abstain
• Verification must be outside the model and proportional to stakes
• Accountability does not delegate
• The capability boundary is jagged, and your impression of it is unreliable

Every one of these was as true three model generations ago as today. Model improvements have changed how often failures occur. They have not changed the kind of thing that fails.

Why the durable half stays durable

Worth understanding rather than taking on trust.

These properties are not deficiencies awaiting a fix. They follow from what the system is: something that produces a probability distribution over text conditioned on a context.

A better model gives you a better distribution. It does not give you retrieval instead of prediction, or unlimited context, or a mechanism that knows what it does not know.

So as long as the basic approach holds, so do these. If something arrives that changes the approach fundamentally, that will be big enough news that you will not miss it.