A model does not see letters or whole words. Text is chopped into tokens: short chunks, usually a few characters, sometimes a whole word. ‘Companion’ might be one token, and ‘unbelievable’ might be three.
Tokens sound like trivia until you notice that everything about using AI is priced and limited in them. Speed is tokens per second. Cost is dollars per million tokens. The size of what a model can consider at once is a token count.
Why tokens cost money
Every token the model reads or writes is real computation on expensive hardware. A long conversation makes the model reread everything so far to produce each new piece, which is why costs climb as chats grow and why providers meter usage in tokens rather than messages.
This is also why free plans have limits, and why an agent left running on its own can quietly rack up a bill. Every step an agent takes, every page it reads, and every retry costs tokens. Good products show you your usage, let you set a budget, and stop to ask before spending real money on a long job.
Why long chats degrade
Models have a maximum number of tokens they can consider at once. When a conversation gets close to that limit, something has to give: the app trims or summarizes older messages. Details you mentioned an hour ago can literally no longer be in front of the model.
You have probably felt this even before hitting any hard limit. As a chat fills up, the model has more and more to weigh every time it answers, and old details start competing with new ones for attention. Answers drift, rules get forgotten, and the sharp assistant from message five gets vague around message eighty. That drift is the token budget straining before it actually breaks.
What to do about it
Long documents and long chats are not free, so spend tokens where they matter. Start fresh conversations for new topics, paste the paragraph that matters instead of the whole file when precision counts, and when an assistant seems to lose the plot, suspect the token budget before you suspect the intelligence.
It also helps to watch what you paste. Dropping in a whole contract to answer a question about one clause spends thousands of tokens on text that does not matter. Pasting the one page you care about, with a line saying what to look for, is cheaper and gets you a more accurate answer at the same time.