AI Concepts·intermediate
Token
The unit the AI actually reads — roughly a word, often less.
AI models don't read characters or whole words. They read tokens — chunks of text, usually 1 to 4 characters.
Rough rules for English:
- 1 token ≈ 3/4 of a word
- 100 tokens ≈ 75 words
- 1,000 tokens ≈ 750 words ≈ 1.5 pages
Common words like "the" are one token. Rarer words get split — "unbelievable" might be un + believ + able.
Why you care: API pricing is per token. Context windows are measured in tokens. Long outputs take longer because the model generates one token at a time.