Back to Glossary
Small Language Model (SLM)
Synonyms: SLM, compact language model, lightweight LLM, small model, tiny LLM
Do not index
Definition
A small language model (SLM) is a language model with a few million to a few billion parameters. Large models have hundreds of billions or more. SLMs cost less and respond faster, and some run on a phone. They also know less and reason less. Model size decides how fast a feature feels and what it costs per use.
Use cases
The wrong size means paying a huge model to tag emails, or shipping a small one that invents facts.
- Overpaying for easy work: Every tagging request goes to the biggest model. NVIDIA researchers estimate a 7B model is 10–30× cheaper to serve than a 70–175B one.
- Small model, big question: An SLM gets asked for facts it never learned, and IBM notes SLM answers need fact-checking.
- Silent quality drops: A router sends a hard question to the small model, and the user gets a shallow answer with no hint a better one existed.
How it's used in practice
- Sort features by task type: Summarizing, classifying, extracting, and rewriting suit small models. Multi-step reasoning goes to large ones.
- Route by difficulty: Default to the small model. RouteLLM cut costs by over 85% on one benchmark while keeping 95% of GPT-4's quality.
- Fine-tune for one job: A specialized SLM can beat a general large model on a narrow task, such as rating product reviews.
- Put the facts in the prompt: Give the small model verified information, as Apple advises for its 3B on-device model.
Add a "Get a deeper answer" button. It reruns the request on the large model. Users get a better answer on demand, and their clicks show where the small model falls short.
Challenges & limitations
- Narrow knowledge: SLMs struggle with broad facts and complex reasoning.
- Router mistakes are hidden: Users can't tell which model answered, so they blame the product.
- Benchmarks mislead: Headline scores rarely predict results on your task. Test with your own data.
Free resources
- Bigger Isn't Always Better (IBM) — pairing small domain models with large cloud models
- 3 Key Features and Benefits of SLMs (Microsoft) — fine-tuning, device fit, and lower latency
- Tiny but Mighty: The Phi-3 Models (Microsoft) — choosing small vs. large models per task, and using an LLM as a router
