Free sample lesson · Fine-Tuning Small Language Models
The cost and latency case
Lesson 1 of the Wondering course Fine-Tuning Small Language Models
Two numbers that change plans
Every model choice comes down to two practical costs: what you pay per call, and how long the user waits.
Small models change both numbers at once. Compared with frontier models, they deliver 25–60x cost savings and sub-second latency.
- Cost savings: Compared with frontier models.
- Practical costs
- Per-call cost: The amount paid for each model call.
- User wait: The time a user waits for a response.
- Small-model effects
- Sub-second latency: Small models deliver responses in less than one second.
- Economic footing: The same work can have a different economic footing.
The useful question is where lower cost matters most.
That is not a small tuning gain. It is a different economic footing for the same work.
The interesting question is not whether small models are cheaper. It is where that cheapness actually matters.
How big is 25–60x?
Multipliers are easy to read past. Put them on a chart and the gap becomes physical.
If a frontier model call costs one unit of budget, the small model equivalent costs somewhere between a fortieth and a twenty-fifth of that.
25x is the conservative end of the savings range. 60x is the strong end.
One call at either end feels trivial. The difference only shows up at volume.
Volume is the multiplier
A 25–60x saving on one call saves almost nothing. The same saving across millions of calls reshapes a budget.
This is why high-volume, repetitive tasks are the natural home for small models. The savings compound with every request.
Latency behaves the same way. Sub-second responses matter most when a person is waiting again and again in a tight loop.
Low-volume, one-off work simply does not generate enough calls for the multiplier to bite.
- High-volume repetitive
- Low-volume one-off
Latency is felt, not billed
Cost shows up on an invoice at the end of the month. Latency shows up in the product immediately.
Sub-second is the threshold where a response stops feeling like a wait and starts feeling like a reply.
In a repetitive workflow, that difference repeats hundreds of times a day for the same person.
So the two benefits are not interchangeable. One protects the budget, the other protects the experience.
- Budget protection
- Experience protection
Making the call
The case for small models is narrow and strong rather than universal.
Start by asking about volume. Repetitive, high-frequency work is where 25–60x cost savings turn into real money.
Then ask about the user. If someone is waiting inside a loop, sub-second latency is the deciding factor.
If neither condition holds, the frontier model premium may still be worth paying.
- Assess workload
- Check volume: Ask whether the work is repetitive and high frequency.
- Check user: Ask whether a person is waiting inside a loop.
- Any condition holds?: A small-model case exists when either condition is present.
- Choose small model: Use the small model when cost savings or sub-second latency matters.
- Consider frontier premium: The frontier model premium may be worth paying.
Check yourself
Compared with frontier models, what combination of benefits do small models provide?
- 25–60x cost savings and sub-second latency
- 2–6x cost savings and one-minute latency
Answer: 25–60x cost savings and sub-second latency
Small models are described as delivering both 25–60x cost savings and sub-second latency compared with frontier models.
The conservative end of the savings range is _____, while the strong end is _____.
Answer: The conservative end of the savings range is 25x, while the strong end is 60x.
The conservative end of the savings range is 25x, while 60x represents the strong end.
Which scenario best illustrates where a 25–60x cost saving is most likely to reshape a budget?
- A repetitive task making millions of requests
- A one-off task making a single request
- A rare task making a few requests each year
- A task whose requests are unrelated and infrequent
Answer: A repetitive task making millions of requests
Savings compound across requests, so millions of repetitive calls can substantially reshape a budget. A one-off call usually does not generate enough volume for the multiplier to matter.
Keep going
This is lesson 1 of 11 in Fine-Tuning Small Language Models. Next up:
- Data privacy as a forcing function
- When small beats frontier
- LoRA: fine-tune in an hour
- Distillation: learning from a larger model
- Quantization: shrinking model size
Wondering turns any topic into short lessons you can read, watch, or listen to, with practice questions and an AI tutor.