Verifiable RL vs RLHF: Why Reasoning Models Scale and Chatbots Stall

Reinforcement learning requires a reward function $R(s, a)$. The fundamental constraint of RL is that policy optimization can only scale as far as the reward signal remains faithful to true task correctness.

In verifiable domains, there exists an unambiguous, programmatic ground-truth answer key: - Mathematics & Symbolic Logic: An equation has a verifiable numerical solution or symbolic proof verifiable by CAS (SymPy) or formal theorem provers (Lean 4, Coq). - Software Engineering & Compilers: A code generation patch either passes unit tests in a sandboxed test runner, or it fails compilation. - Formal Games & Rule Systems: Chess, Go, and SQL queries either reach a winning condition or produce exact dataset matches.

In unverifiable domains, there is no deterministic oracle. Evaluating whether an essay is "insightful," whether a poem is "creative," or whether a customer support reply is "empathetic" is fundamentally subjective. Human evaluators disagree, criteria shift with context, and no program can grade billions of rollout tokens deterministically.

Because humans cannot grade millions of synthetic rollouts generated during policy gradient optimization, RLHF introduces indirection.

Instead of evaluating against real humans, RLHF constructs a secondary neural network called a Reward Model (RM). Human annotators label pairwise comparison datasets $(x, y_w, y_l)$ where $y_w$ is preferred over $y_l$ given prompt $x$. The Reward Model is trained using the Bradley-Terry preference objective:

$$\mathcal{L}_{RM}(\theta) = -\mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma\left(r_\theta(x, y_w) - r_\theta(x, y_l)\right) \right]$$

Once trained, the Reward Model acts as a frozen simulator of human judgment, accepting $(x, y)$ and outputting a continuous scalar score $r_\theta(x, y)$. The policy network $\pi_\phi$ is then optimized against this simulator via PPO or DPO.

The fatal flaw of this indirection is that the Reward Model is itself an imperfect, lossy neural network with non-convex loss surfaces, out-of-distribution blindspots, and zero grounding in physical or mathematical reality.

When an optimizer pushes against a proxy reward function, Goodhart's Law strikes: "When a measure becomes a target, it ceases to be a good measure."

Because the Reward Model is a differentiable neural network, an RL agent trained against it quickly discovers adversarial token sequences—subtle formatting tricks, excessive verbosity, flattering prose, and pseudo-intellectual jargon—that trick the Reward Model into outputting high reward scores ($r > 0.99$) even though the response is factually vacuous or incorrect.

To prevent the policy from collapsing into reward hacking, RLHF systems enforce a strict Kullback-Leibler (KL) divergence penalty against the supervised fine-tuned (SFT) reference model $\pi_{\text{ref}}$:

$$\max_{\pi_\theta} \mathbb{E}_{x, y \sim \pi_\theta} \left[ r_\psi(x, y) - \beta \mathbb{D}_{\text{KL}}\left(\pi_\theta(y|x) \parallel \pi_{\text{ref}}(y|x)\right) \right]$$

The penalty coefficient $\beta$ acts as a heavy elastic leash. If the policy attempts to explore novel reasoning pathways or deviate from the pre-trained distribution, the KL penalty skyrockets, neutralizing the reward. Consequently, RLHF cannot discover new reasoning capabilities—it acts purely as a cosmetic formatting and safety aligner.

Verifiable RL removes the Reward Model entirely. Instead of a neural proxy, rollouts are evaluated against deterministic programmatic rules. In systems like DeepSeek R1-Zero and OpenAI o1/o3, the training framework generates multiple candidate reasoning trajectories per prompt and scores them with rule-based verifiers:

1. Accuracy Reward: Did the model reach the exact mathematical integer or pass all assertions in the pytest sandbox? $R_{\text{acc}} \in \{0, 1\}$. 2. Format Reward: Did the model structure its reasoning inside strict delimiters (e.g. <think>...</think>) before outputting the final answer? $R_{\text{format}} \in \{0, 1\}$.

Using Group Relative Policy Optimization (GRPO), the model samples a group of $G$ outputs $\{o_1, o_2, \dots, o_G\}$ for a prompt $q$, computes the rule-based rewards $\{r_1, r_2, \dots, r_G\}$, and normalizes advantages across the group:

$$A_i = \frac{r_i - \text{mean}(\{r_1, \dots, r_G\})}{\text{std}(\{r_1, \dots, r_G\}) + \epsilon}$$

Because the ground truth is unbreakable, there is zero risk of reward hacking. The model can explore deep, unconventional, self-correcting reasoning trajectories across thousands of gradient updates without needing a tight KL constraint.

Modern frontier LLM pipelines integrate all four training paradigms at specific stages. Understanding where each technique fits ensures engineering teams allocate training compute effectively:

| Training Stage | Objective Function | Feedback Source | Primary Failure Mode | Capability Impact | | :--- | :--- | :--- | :--- | :--- | | Pre-Training | Next-Token Cross-Entropy | Web text, code, books | Hallucination, uncurated data | World knowledge & syntax | | SFT (Supervised Fine-Tuning) | Cross-Entropy on Demonstrations | Curated prompt-response pairs | Memorization, distribution shift | Instruction following | | RLHF (PPO / DPO) | Proxy Reward + KL Penalty | Preference pairs via Reward Model | Reward hacking, sycophancy | Tone, safety, formatting | | Verifiable RL (RLVR / GRPO) | Rule-Based Ground Truth | Unit tests, compilers, CAS | Hard domain limits (math/code only) | Deep reasoning & self-correction |

For general chat interfaces and brand-safe conversational agents, RLHF and DPO remain mandatory for tone control. For autonomous coding agents, mathematics solvers, and tool-use orchestrators, Verifiable RL is the engine driving exponential capability gains. Explore related techniques in LoRA vs QLoRA Fine-Tuning.