Token
A token is a unit of information that a language model processes as part of its input or output.
Tokens do not correspond exactly to words.
A word may be represented by one token, several tokens, or as part of a token depending on the model's tokenizer.
Why Models Use Tokens
Language models operate on tokenized representations rather than directly processing written text as whole words.
A simplified process is:
Text
↓
Tokenization
↓
Tokens
↓
ModelGenerated output is also produced as tokens and then represented as readable text.
Tokens and Cost
Many AI APIs measure usage partly according to:
- Input tokens
- Output tokens
This means longer prompts and longer outputs can affect cost.
OpenAI provides token-counting tooling specifically for understanding context usage and estimating request size. OpenAI token-counting guide
Tokens and Context
The Context Window is usually described in tokens.
The complete input may include:
- Prompt
- Conversation history
- Retrieved documents
- Tool definitions
- System instructions
- Images or other encoded inputs
- Generated output allowance
That means adding more information always has some context cost.
Why It Matters
Developers sometimes treat context as unlimited because models can accept very large inputs.
But additional tokens can affect:
- Cost
- Latency
- Context management
- Retrieval strategy
- Available output length
More context is not automatically better context.
Common Problems
Common problems include:
- Estimating tokens as exact word counts
- Sending full documents when only a few sections are relevant
- Repeating the same instructions
- Conversation history growing indefinitely
- Large tool schemas consuming context
- Ignoring output-token requirements
How to Improve
Send information because it is relevant, not merely because it fits.
Use Retrieval-Augmented Generation and other context-management techniques to select useful information instead of continually expanding the prompt.