Back to Glossary
Long-Context Modeling
Synonyms: long context window, large context window, million-token context, extended context
Do not index
Definition
Long-context modeling is an AI model's ability to read and use very long inputs in one request, from a 400-page contract to a weeks-long chat thread. Leading models now take up to about 1 million tokens. For UX, it shapes what users can upload, how the product shows what the model is working from, and what happens near the limit. It exists because people bring whole documents, and splitting them loses the links between parts.
Use cases
Long context fails quietly, and answers still sound confident.
- The buried clause: A legal team uploads a 400-page contract. The summary skips an indemnity clause on page 210. Models recall the start and end of long inputs better than the middle.
- Silent forgetting: 3 hours into a chat, the assistant breaks a rule set in message 1. Early messages got summarized without telling the user.
- The 1M-token promise: Marketing says "upload your whole codebase." Answers get worse as the upload grows.
How it's used in practice
- Show what the model sees: List attached files and a usage meter, in pages or messages.
- Warn before the limit: "Older messages will be summarized. Pinned rules stay."
- Let users pin instructions: Key rules ride along in every turn.
- Cite page numbers: Answers quote the passage and its location so users can check.
- Offer a fresh start: A new chat, seeded with a handoff summary.
Find your real upload limit with a planted fact. Hide one made-up detail halfway through your longest real document, then ask about it. Repeat at 3 or 4 file sizes. Set the upload cap just below the size where answers start missing it.
Challenges & limitations
- Cost and speed: Every page rides along in each request, so long threads get slower and pricier.
- Tokens mean nothing to users: Translate them into pages or messages.
Free resources
- Chroma: Context Rot — tests on 18 models showing reliability drops unevenly as inputs get longer.
- GitHub: chroma-core/context-rot — code to reproduce Chroma's long-input experiments.
- Anthropic: Prompting Claude's long context window — quote extraction before answering cut long-document errors by 36% for Claude 2.
