Token
A token is a chunk of text a language model processes at once—roughly a word or part of a word—and is the unit used for pricing and context limits.
In Simple Terms
Think of it as the smallest Lego piece the model can see or produce; pricing is per piece.
Detailed Explanation
Models do not read whole words; they split text into tokens. One token is often a word or subword, so token count is usually close to word count but not identical. APIs bill and limit by tokens. When it matters: when budgeting cost, designing prompts, or fitting content into a context window. Common mistakes: assuming one word equals one token, or ignoring token limits when sending long documents.
Related Terms
Transformer
A transformer is a neural network architecture that uses attention to process sequences (e.g., text or tokens) in parallel rather than step-by-step. It underlies most large language models and many vision and multimodal systems.
Read moreAttention Mechanism
The attention mechanism lets a model focus on different parts of its input when producing each part of the output. It is the core of transformer architectures and enables handling long sequences and rich context.
Read moreFine-Tuning
Fine-tuning is training a pre-trained model on your own data so it gets better at specific tasks or styles while keeping its general abilities.
Read moreWant to Implement AI in Your Business?
Let's discuss how these AI concepts can drive value in your organization.
Schedule a Consultation