What AI actually computes

Two jobs inside one chatbot

A chatbot does two very different jobs. Mixing them up causes confusion about the machines behind it.

Training changes the model's weights, the learned numbers it uses. It learns from examples.

Inference uses those weights to produce an output, such as a reply to your question.

So when a chatbot answers you, it is usually doing inference. The learning happened earlier.

One neuron, one small sum

A model is built from many numerical transformations. Each one is mostly multiplication and addition.

Picture one neuron with inputs 2 and 3, and weights 0.4 and 0.7.

It computes $$2 \times 0.4 + 3 \times 0.7 = 2.9$$, then applies another function to that result.

Real models repeat steps like this a huge number of times. Hardware exists to run these sums.

How training changes a weight

Training starts by comparing a prediction with a target, the answer the model should have given.

Backpropagation calculates how the error depends on each weight. An optimizer then uses that information to update the weights.

In our neuron, replacing 0.4 with a new value after that comparison is a training update.

Then the model predicts again, and the cycle repeats over many examples.

Answering does not retrain

A common belief: every chatbot answer retrains the whole model. It does not.

Reusing 0.4 and 0.7 on new inputs is inference. The weights stay the same.

One request may run many inference steps, yet update no weights at all.

Training and inference also have different resource demands, even when they run on the same chip.

Chip, model, algorithm

Three terms often get blurred together, but they name different kinds of things.

A GPU is hardware. A transformer is a model architecture. Backpropagation is a learning algorithm that works out how error depends on weights.

Hardware executes the operations. Sze and colleagues survey the hardware needed for neural-network calculations.[[1]](https://arxiv.org/abs/1703.09039)

Three meanings of compute

The word compute hides three different ideas.

It can mean an amount of work. A FLOP is one floating-point operation, so counting FLOPs measures work done.

It can mean a rate. FLOP/s counts work per second, often quoted as a peak.

It can mean access to resources. A GPU-hour rents one processor for an hour, without saying how much useful work it completes.

Reading hardware claims wisely

Neither FLOPs nor FLOP/s counts intelligence. They measure arithmetic only.

So more peak FLOP/s does not guarantee a better answer. Answer quality depends on the model and its learned weights.

Renting an unspecified GPU for an hour says little about how many predictions you will get.

When you meet a claim, ask: is this training, inference, or rented capacity?

Replacing a model weight after comparing a prediction with its target is training.

• True
• False

Answer: True

Training changes learned weights after the model learns from examples. Inference uses existing weights to produce an output without changing them.

Compute can mean an amount of work measured in _____, a processing rate measured in _____, or access to resources rented as _____.

Answer: Compute can mean an amount of work measured in FLOPs, a processing rate measured in FLOP/s, or access to resources rented as GPU-hours.

FLOPs count arithmetic work, FLOP/s measures how quickly work can be performed, and GPU-hours describe rented processor time without specifying completed work.

A vendor says, “This GPU has higher peak FLOP/s, so it will always produce better chatbot answers.” Which response is most accurate?

• Correct, because arithmetic rate directly determines answer quality.
• Correct, because every faster GPU has a more capable model.
• Incorrect, because answer quality depends on the model and its learned weights.
• Incorrect, because FLOP/s measures the number of rented GPU-hours.

Answer: Incorrect, because answer quality depends on the model and its learned weights.

Peak FLOP/s measures a maximum arithmetic rate, not intelligence or answer quality. Quality also depends on the model and its learned weights. A higher rate may help performance, but it does not guarantee better answers.