- The largest general-purpose model is rarely the cheapest or most accurate choice for a narrow task.
- Small, fine-tuned models running on modest hardware now match frontier models on well-scoped jobs.
- Teams that route by task, not by habit, cut inference cost sharply without a quality drop.
For a while, the default engineering decision was simple: call the biggest, newest model available and let it handle everything from summarizing a support ticket to writing production code. That default is quietly breaking down, not because the big models got worse, but because teams finally started measuring what each task actually needs, and most tasks need far less than they were getting.
The gap between capability and requirement
Classifying a support ticket, extracting a field from an invoice, or tagging a log line does not require a model that can reason about physics or write a novel. It requires consistent, fast, cheap judgment on a narrow input. A small model fine-tuned on a few thousand examples of exactly that task now matches or beats a frontier model on accuracy, while running for a fraction of the cost and a fraction of the latency.
Where the frontier model still earns its keep
None of this argues for abandoning large models. Open-ended reasoning, unfamiliar problems, and anything where the input distribution cannot be anticipated in advance still favor a general-purpose model with broad training. The mistake was never using large models. It was using only large models, for every call, regardless of the task.
| Task type | Better fit | Why |
|---|---|---|
| Open-ended reasoning, unfamiliar input | Large frontier model | Needs broad knowledge and flexible reasoning |
| Classification, extraction, tagging | Small fine-tuned model | Narrow, repeatable task; small model matches accuracy at a fraction of cost |
| High-volume, latency-sensitive calls | Small model, often on-device | Cost and speed compound at scale; local inference avoids network round-trips |
What routing by task actually looks like
Teams seeing the biggest savings built a simple router in front of their model calls: known, narrow tasks go to a small fine-tuned model; anything novel or ambiguous escalates to the larger model. The router itself is unglamorous, a classifier or even a rule set, but it is where most of the cost reduction comes from. It is the same discipline behind the edge-versus-cloud split we covered in why edge computing is replacing the cloud-first default: match the workload to the resource it actually needs.
Frequently asked questions
Are small language models actually as accurate as large ones?
On narrow, well-scoped tasks with enough fine-tuning examples, yes, often within a
small margin. On open-ended or unfamiliar tasks, large models still lead.
How much can task-routing actually save on inference cost?
Teams routing high-volume, narrow tasks to small models commonly report cost cuts of
more than half, since small models are cheaper per call and often fast enough to run
on local or edge hardware entirely.