Context Window
A context window is the amount of information an AI model can consider within one model interaction or active context.
It is commonly measured in tokens.
What Can Consume Context
The context may include:
- System instructions
- Prompt
- Conversation history
- Retrieved documents
- Tool definitions
- Function results
- Uploaded content
- Model output
The exact accounting depends on the model and API.
Context Is Not Long-Term Memory
A large context window does not mean the model permanently remembers everything placed inside it.
Context is the information currently made available to the model for the interaction.
Longer-lived application state generally needs to be stored elsewhere and selectively provided again when relevant.
Larger Is Not Always Better
A large context window makes longer inputs possible.
That can be useful for:
- Long documents
- Extended conversations
- Large codebases
- Multiple retrieved sources
But sending the maximum possible context can increase:
- Cost
- Latency
- Irrelevant information
- Difficulty identifying the most important material
Retrieval-Augmented Generation often exists partly to select the most relevant information rather than providing an entire knowledge base every time.
Why It Matters
The context window creates a practical boundary around what the model can directly consider.
Application architecture therefore needs to decide:
What information belongs in this interaction?
not:
What information do we possess?
Common Problems
Common problems include:
- Entire database inserted into every prompt
- Old conversation retained indefinitely
- Important instructions competing with irrelevant text
- Context window mistaken for persistent memory
- No strategy for long documents
- Retrieval results consuming too much space
How to Improve
Prioritize relevant context.
Use:
- Retrieval
- Summarization
- Structured state
- Selective history
- Appropriate tool calls
instead of treating the context window as general-purpose storage.