How LLMs Work: Tokens, Context & Sampling
An agent is an LLM in a loop, using tools. So before you build one, you need to understand the engine it drives: what a large language model actually does, what it can hold in mind, and how you steer its output. Everything later — tool use, memory, agent loops — rests on these facts.
An LLM predicts the next token
At its core, a large language model does one thing: given some text, it predicts a probability distribution over the next token, picks one, appends it, and repeats. This is autoregressive generation — the whole response is produced one token at a time, each conditioned on everything before it. There is no plan and no database lookup; there is next-token prediction, over and over.
Tokens
Models don''t see characters or words — they see tokens, the subword pieces text is split into (roughly 4 characters, or about three-quarters of a word, each). "tokenization" might be two tokens; a rare word several. This matters because everything is measured and billed in tokens: prompt length, output length, limits, and cost.
The context window
The context window is the maximum number of tokens a model can consider at once — the input prompt and generated output together. Anything outside it simply doesn''t exist for the model. A short chat fits easily; a long agent conversation or a big document does not. This single limit is why agents need Memory Systems and RAG (later modules): to summarise, store, and retrieve information that won''t fit in the window.
Sampling and temperature
The model gives probabilities; sampling turns them into an actual choice. Temperature controls the randomness:
- Low ($\approx 0$) — nearly deterministic, almost always taking the most likely token. Focused and repeatable — usually what you want for tool-calling agents.
- High — flatter odds, more varied and "creative" output, at the cost of reliability.
(top_p is a related knob.) For agents that must emit exact tool calls and valid JSON, low temperature is the norm.
The model is stateless
Each API call is independent — the model remembers nothing between calls. What looks like a "conversation" is just the entire message history resent every time. Agents are built on this fact: the loop re-sends the growing transcript (plus tool results) on every step.
Common pitfalls
- Context limits. You can''t stuff unlimited history in; past a point you must summarise or retrieve (memory/RAG).
- Hallucination. The model predicts plausible tokens, not true ones. It can state falsehoods fluently — so agents ground answers with tools and retrieval and verify them.
- Determinism. Only temperature $0$ is (mostly) repeatable; higher settings vary run to run.
- Token cost. Long prompts and histories cost tokens (money and latency) — trim what you send.
Why this is your on-ramp
Agent Foundations, Agent Loops, and Memory Systems all assume you know what a token is, what the context window constrains, why the model is stateless, and how temperature affects reliability. Nail these and the rest of the course is engineering, not mystery.