🤖 LLM Internals — Claude, Claude Code, Tokens & the Request/Response Cycle¶
How LLMs like Claude work under the hood, from prompt assembly to tokenization, inference, sampling, tool-use loops and cost. Claude- and Claude Code–specific claims are checked against Anthropic's official docs (October 2026); general transformer mechanics are labelled as such, because Anthropic doesn't publish Claude's architecture.
📦 Contents¶
| # | Document | Description |
|---|---|---|
| 1 | How Claude Works | What's public vs not, transformer background, training (pre-training, RLHF, Constitutional AI), context window, prefill/decode, sampling, safety |
| 2 | Claude Code — Interaction Flow | The agentic harness: what's loaded into context, real tool names, the request/stream/tool loop, context management, permissions |
| 3 | The Request/Response Cycle | Messages API request anatomy, what happens server-side, streaming events, stop reasons, token growth across tool calls |
| 4 | Tokenization & Token Calculation | BPE, counting tokens with the API, input/output/cache/batch pricing, prompt caching mechanics |
| 5 | System Prompt Engineering | Anthropic's current prompting guidance, patterns, antipatterns, and evaluating prompts |
| 6 | How Claude Makes Code Changes | Traced code-change and debugging loops: Read/Edit/Write/Bash, permissions, hooks, checkpoints |
| 7 | Cost Optimization Best Practices | The cost levers in order of payoff, measurement, and Claude Code habits |
🎯 Why This Matters¶
| Reason | Impact |
|---|---|
| Cost | Token-aware design (caching, batching, context hygiene) is usually the biggest lever on an LLM bill |
| Latency | Prefill drives time-to-first-token; decode drives tokens per second |
| Prompt quality | Knowing what the model actually sees explains most "weird" behaviour |
| Debugging | Tokenization explains character-level failures and surprising token counts |
| Production | Capacity planning, rate limits, caching and batching strategies |
| Agent design | Tool-use loops are just repeated requests with growing context |
🏗️ Architecture Overview¶
┌──────────────────────────────────────────────────────────────────────────┐
│ ONE LLM REQUEST, END TO END │
│ │
│ USER: "Write a function to calculate Fibonacci numbers in Python" │
│ │ │
│ ▼ │
│ PROMPT ASSEMBLY (client) tools → system → messages │
│ tool defs system prompt history + tool results new message │
│ │ │
│ ▼ │
│ TOKENIZATION (server) text → token IDs (model-specific tokenizer) │
│ e.g. ~1,200 input tokens; count exactly with count_tokens │
│ │ │
│ ▼ │
│ INFERENCE │
│ 1. Prefill: all input tokens in parallel → KV cache │
│ (cached prefix reused if prompt caching hits) │
│ 2. Decode: one token per step, sampled from the next-token │
│ distribution (shaped by effort/thinking on current models) │
│ 3. Stop: end_turn | tool_use | max_tokens | stop_sequence | refusal │
│ │ │
│ ▼ │
│ RESPONSE content blocks (thinking, text, tool_use) + stop_reason │
│ + usage (input, cache read/write, output tokens) │
│ "def fibonacci(n):\n if n <= 1:\n return n ..." │
└──────────────────────────────────────────────────────────────────────────┘
🚀 Quick Links¶
| Topic | Read This First |
|---|---|
| New to LLMs? | How Claude Works |
| Using Claude Code? | Claude Code Interaction |
| Building agents? | Request/Response Cycle |
| Optimizing costs? | Cost Optimization Best Practices (start here) or Tokenization & Cost (deep dive) |
| Writing prompts? | System Prompt Engineering |
🔗 Related Modules¶
- AI Agents — How agents use LLMs in loops with tool calls
- MCP Protocol — How MCP connects AI applications to tools
- RAG Pipeline — How retrieval augments LLM knowledge
- Harness & Loop Engineering — Building the scaffolding around the model