LLM Internals: Pretraining, Tokenization, SFT, and RLHF

Pretraining forms the foundational worldview and parametric memory of an LLM. Training a frontier model requires trillions of tokens (typically 3 to 15 trillion tokens) extracted from web crawls, open-source code repositories, books, and scientific papers.

The 4-Stage Data Ingestion Pipeline

1. Massive Web Extraction (Common Crawl & Custom Scrapers): Raw HTML is parsed, extracting raw text while stripping boilerplate (navigation bars, ads, tracking scripts). FastText classifiers filter out non-target languages and low-quality machine-translated spam.

2. Deduplication at Scale (MinHash & Locality-Sensitive Hashing): Up to 30% of the public web consists of duplicate or near-duplicate documents. Document-level and paragraph-level deduplication using MinHash LSH and $n$-gram matching prevents models from memorizing repetitive web boilerplate and wasting training FLOPs.

3. Heuristic & Toxicity Filtering: Gopher quality heuristics filter out documents with extreme character-to-word ratios, low unique token counts, or excessive ellipsis punctuation. Model-based classifiers scrub personally identifiable information (PII), toxic text, and copyrighted benchmark test sets (to prevent contamination).

4. Synthetic Data & Curriculum Upsampling: High-quality domains (Python source code, math proofs, structured textbooks) are upsampled 2× to 5× relative to raw web text. Synthetic reasoning traces generated by teacher models augment sparse domains like formal logic and code generation.

Neural networks cannot process raw character strings directly; they operate on continuous vector embeddings. The bridge between text and vectors is the Tokenizer, typically implemented via Byte-Pair Encoding (BPE) or Byte-level BPE (BBPE).

The Byte-Pair Encoding Algorithm

BPE builds a fixed vocabulary (typically 32,000 to 128,000 tokens) by iteratively merging the most frequently occurring adjacent byte or character pairs:

1# Conceptual Byte-Pair Encoding (BPE) Merge Step2from collections import Counter3 4def get_stats(vocab: dict[str, int]) -> Counter:5    pairs = Counter()6    for word, freq in vocab.items():7        symbols = word.split()8        for i in range(len(symbols) - 1):9            pairs[symbols[i], symbols[i + 1]] += freq10    return pairs11 12def merge_vocab(pair: tuple[str, str], v_in: dict[str, int]) -> dict[str, int]:13    v_out = {}14    bigram = " ".join(pair)15    replacement = "".join(pair)16    for word in v_in:17        w_out = word.replace(bigram, replacement)18        v_out[w_out] = v_in[word]19    return v_out

Inherent Token-Level Blindspots

Because the model receives integer token IDs rather than raw letters, several notorious failure modes emerge:

• Letter Counting ("How many 'r's in strawberry?"): In BPE, "strawberry" is tokenized as ["str", "aw", "berry"]. The model never observes the individual characters r, r, r in its input tensor.
• Arithmetic Failures: Multi-digit numbers (e.g., 18492) are unpredictably split into arbitrary token chunks (e.g., ["18", "492"] vs ["184", "92"]), breaking spatial alignment for standard columnar addition.
• Whitespace & Case Sensitivity: Adding a leading space (" Hello" vs "Hello") produces completely distinct token IDs, causing unexpected shifts in probability distributions.

LLM inference is divided into two distinct computational phases, each bounded by different hardware constraints on the GPU:

1. The Prefill Phase (Compute-Bound) Ingests the entire prompt sequence simultaneously. Computes Query ($Q$), Key ($K$), and Value ($V$) projections for all prompt tokens in parallel using high-performance matrix-matrix multiplication (GEMM). Hardware Bottleneck:* Arithmetic throughput (TFLOPs) of the GPU tensor cores.

2. The Decode Phase (Memory-Bandwidth Bound) Generates tokens sequentially, one token per forward pass. To avoid recalculating attention keys and values for all past tokens at every step, past Key and Value vectors are stored in GPU High-Bandwidth Memory (HBM) as the KV Cache. Hardware Bottleneck:* GPU Memory Bandwidth (GB/s). The GPU must stream gigabytes of KV cache tensors from VRAM for every single token generated.

1# KV Cache VRAM Formula per Token:2# Memory (Bytes) = 2 * 2 * n_layers * n_heads * d_head * seq_len * batch_size3# (Factor of 2 for K and V, factor of 2 for FP16 16-bit floats)4 5# Example: 70B Model (80 layers, 64 heads, 128 dim), 8k Context, Batch 1:6# Memory = 2 * 2 * 80 * 64 * 128 * 8192 * 1 ≈ 21.4 GB VRAM just for KV Cache!

Inference Optimizations

• PagedAttention (vLLM): Allocates KV cache memory dynamically in fixed-size virtual pages, eliminating internal fragmentation and boosting serving throughput by 2×–4×.
• FlashAttention: Fuses softmax and attention matrix operations into GPU SRAM, reducing memory read/write overhead from $O(N^2)$ to $O(N)$.
• Speculative Decoding: Uses a small, fast draft model to speculate $K$ tokens in parallel, which the large target model verifies in a single forward pass.

Where does an LLM store knowledge? Modern transformer architectures decouple static world knowledge from active conversational context:

Parametric Weights (Long-Term Storage) Knowledge is stored implicitly across billions of static floating-point weights in the Feed-Forward Networks (MLP layers). MLPs act as key-value associative memories: early transformer layers detect syntactic patterns, while deeper MLP layers trigger factual associations (e.g., "Eiffel Tower" $ ightarrow$ "Paris"). Drawback:* Knowledge is fixed at training time, prone to hallucination under low probability distributions, and impossible to update without expensive fine-tuning.

Working Memory (In-Context Activations) Working memory consists of active activations and KV vectors maintained within the attention context window. Attention heads dynamically route, attend to, and synthesize facts provided in the prompt. Mechanism:* In-Context Learning (ICL) allows models to execute tasks, adapt to new schemas, and reference fresh facts without changing any underlying model weights.

Retrieval-Augmented Generation (RAG) bridges this divide by retrieving exact external records and injecting them directly into working memory.

A pretrained base model is simply a document predictor: given a prompt like "Write a Python script to sort an array", it might respond by continuing with " 2. Write a function to reverse a string 3. ..." (mimicking a programming homework handout).

Post-Training transforms this document predictor into an aligned conversational assistant through Supervised Fine-Tuning (SFT).

Structuring Multi-Turn Dialogues (ChatML)

SFT wraps unstructured text into structured conversation templates using dedicated delimiter tokens:

1<|im_start|>system2You are a senior backend architect. Respond concisely with verified code.<|im_end|>3<|im_start|>user4How do I implement a token bucket rate limiter in Redis?<|im_end|>5<|im_start|>assistant6Use a Redis Lua script to atomically update timestamps and tokens...<|im_end|>

The Mechanics of Loss Masking

During SFT, the model must learn only how to generate high-quality assistant replies, not memorize the user prompt. To achieve this, training pipelines apply Loss Masking:

1# SFT Cross-Entropy Loss Masking Implementation2import torch3import torch.nn as nn4 5def compute_sft_loss(6    logits: torch.Tensor,       # Shape: (batch, seq_len, vocab_size)7    labels: torch.Tensor,       # Shape: (batch, seq_len)8    prompt_lengths: list[int]   # Token count of system + user prompt9) -> torch.Tensor:10    # Set target label to -100 (ignored by PyTorch CrossEntropyLoss) for prompt tokens11    masked_labels = labels.clone()12    for i, p_len in enumerate(prompt_lengths):13        masked_labels[i, :p_len] = -10014 15    loss_fn = nn.CrossEntropyLoss(ignore_index=-100)16    17    # Shift logits and labels for autoregressive prediction18    shift_logits = logits[..., :-1, :].contiguous()19    shift_labels = masked_labels[..., 1:].contiguous()20    21    return loss_fn(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))

Loss masking ensures gradients are backpropagated exclusively through the assistant response tokens, optimizing conversational alignment without corrupting prompt comprehension.

While SFT teaches models format and tone, it suffers from exposure bias and cannot easily penalize hallucinations or subtle errors. Reinforcement Learning Alignment aligns the model with human preferences for helpfulness, accuracy, and safety.

1. RLHF with PPO (Two-Stage Pipeline)

Step 1 (Reward Model Training): Human annotators rank model completions ($y_w succ y_l$). A Reward Model $r_ heta(x, y)$ is trained using the Bradley-Terry preference objective: $$mathcal{L}_{ ext{RM}} = -mathbb{E}_{(x, y_w, y_l)} left[ log sigma left( r_ heta(x, y_w) - r_ heta(x, y_l) ight) ight]$$ Step 2 (PPO Policy Optimization): The base LLM acts as the policy network, generating completions scored by the Reward Model. A KL-divergence penalty against the original SFT reference model prevents the policy from drifting into degenerate output loops ("reward hacking").

2. Direct Preference Optimization (DPO)

DPO eliminates the complex, unstable PPO training loop (which requires maintaining four large models simultaneously: Actor, Critic, Reference, and Reward Model). By reparameterizing the Bradley-Terry objective, DPO optimizes the policy directly on preference pairs:

$$mathcal{L}_{ ext{DPO}}(pi_ heta; pi_{ ext{ref}}) = -mathbb{E}_{(x, y_w, y_l)} left[ log sigma left( eta log rac{pi_ heta(y_w|x)}{pi_{ ext{ref}}(y_w|x)} - eta log rac{pi_ heta(y_l|x)}{pi_{ ext{ref}}(y_l|x)} ight) ight]$$

DPO delivers stable, closed-form preference alignment with a standard cross-entropy loss function, drastically simplifying the post-training infrastructure.

The final frontier of LLM capability is the transition from passive text generation to active environmental reasoning and tool execution.

Tool Use & Function Calling Mechanics

Function calling does not require special hardware; it relies on structured schema conditioning:

1. Schema Injection: The system prompt injects JSON Schema definitions of callable tools. 2. Constrained Generation: When a query requires external action, the model outputs a structured tool invocation token sequence (e.g., <tool_call>{"name": "fetch_user", "args": {"id": 42}}</tool_call>). 3. Environment Interception: The client harness parses the call, executes the local code/database query, and feeds the resulting output back into the model context as a <tool_response> turn.

1┌─────────────────────────────────────────────────────────────┐2│                    THE ReAct TOOL LOOP                      │3│                                                             │4│   User Prompt ──> LLM Thought ──> Tool Call (JSON)          │5│                                         │                   │6│   LLM Final Answer <── Tool Result <────┘                   │7└─────────────────────────────────────────────────────────────┘

Reasoning Models & Test-Time Compute

Frontier reasoning models (OpenAI o1/o3, DeepSeek-R1) shift compute from pretraining to inference time:

• Chain-of-Thought Search: Models generate thousands of intermediate "thinking tokens" evaluating alternative hypotheses, testing edge cases, and correcting earlier false assumptions before emitting the final answer.
• Verifiable Reward Reinforcement Learning (RLVR / GRPO): In domains with deterministic oracles (math, compiler verification, formal logic), models are reinforced using rule-based rewards ($R in {0, 1}$) rather than human preference models.
• Test-Time Scaling Law: Expanding the number of reasoning tokens at test-time linearly improves accuracy on complex algorithmic tasks without retraining model weights.