What is Tokens?
AI EngineeringLast updated:
The discrete units of text that LLMs read and generate, typically subwords or word fragments.
Tokens are the atomic units of language model processing. A single English word usually maps to 1-3 tokens depending on the tokenizer. Token counts determine API pricing, context window usage, and processing speed.