Economy & Market Auto-verified translation
CONCEPT
token
The base unit of generative AI processing and cost — a short chunk of text, image, or audio a model reads or produces, roughly a syllable of text. Pricing, context limits, and spend are counted in tokens, which makes it an emerging economic unit: as its cost falls, what is worth generating changes.
Origin
From natural-language processing, where text is split into tokens; it became an economic and pricing unit with the rise of commercial large language models.
- attribute
- New unit of measurement for technology spending
- definition
- Economic unit of generative AI — a number between 0 and ~65,000 representing a few letters, pixels, or sound. 1 token ≈ one syllable. 100 tokens ≈ one line of code