Bias, Variance, and Consistency of RL Estimators in LLM Post-Training

Preface

Easily my favorite class during the Fall 2024 semester during my MSML days at CMU was the Intermediate Statistics class by Prof. Larry Wasserman (coded 36-705). That course was about estimators, asymptotic theory, nonparametrics, causal inference, and concentration inequalities. But what I think it was actually about, in hindsight, was a habit of mind: whenever someone hands you a quantity computed from samples, you should immediately ask three questions. Is it unbiased? What is its variance? Is it consistent, and at what rate? (all our assignments were around these xD).

As I see it, the modern practice of post-training large language models with reinforcement learning turns out to be a spectacular playground for exactly these questions. Every algorithm in the alphabet soup of PPO, GRPO, and RLOO is, at its core, a Monte Carlo estimator of a policy gradient, and the differences between them are almost entirely differences in how they trade bias for variance. My goal with this blogpost is to make that statement precise. The blog assumes a background in Statistics and Linear Algebra, though some potentially unfamiliar concepts have been described, either in the next section or marked with a superscript and explained in the sidenotes.

1. Asymptotic Statistical preliminaries

While the best reference is obviously Prof. Larry's book, All of Statistics, this section collects the definitions and limit theorems that the rest of the blog uses, stated for a vector-valued estimator since gradients are vectors. Let $T_G \in \mathbb{R}^d$ be an estimator built from $G$ samples, targeting a fixed vector $g \in \mathbb{R}^d$.

Definition (bias, variance, MSE). The bias is $\mathrm{Bias}(T_G) = \mathbb{E}[T_G] - g$, the variance is $\mathrm{tr}\,\mathrm{Cov}(T_G) = \mathbb{E}\|T_G - \mathbb{E}[T_G]\|^2$, and the mean squared error is $\mathrm{MSE}(T_G) = \mathbb{E}\|T_G - g\|^2$.

Lemma 1 (MSE decomposition). $\mathrm{MSE}(T_G) = \|\mathrm{Bias}(T_G)\|^2 + \mathrm{tr}\,\mathrm{Cov}(T_G)$.

The rest of this blog is about limits of random quantities, so we fix notation for the four standard modes of convergence, decorated as superscripts on the arrow in the style of the textbooks. Plain arrows $\to$ are reserved for deterministic sequences.

Definition (modes of convergence). Let $T_1, T_2, \dots$ and $T$ be random vectors on a common space.

• $T_G \xrightarrow{\text{a.s.}} T$ (almost surely) if $\mathbb{P}\left( \lim_{G \to \infty} T_G = T \right) = 1$.
• $T_G \xrightarrow{P} T$ (in probability) if $\mathbb{P}\left( \|T_G - T\| > \varepsilon \right) \to 0$ for every $\varepsilon > 0$.
• $T_G \xrightarrow{qm} T$ (in quadratic mean) if $\mathbb{E}\|T_G - T\|^2 \to 0$.
• $T_G \xrightarrow{D} T$ (in distribution) if $\mathbb{E}[f(T_G)] \to \mathbb{E}[f(T)]$ for every bounded continuous $f$.

The implications run $\xrightarrow{\text{a.s.}} \Rightarrow \xrightarrow{P}$, $\xrightarrow{qm} \Rightarrow \xrightarrow{P}$, and $\xrightarrow{P} \Rightarrow \xrightarrow{D}$, with none of the arrows reversible in general, and with almost sure and quadratic mean convergence incomparable to each other.

Definition (consistency). $T_G$ is consistent for $g$ if $T_G \xrightarrow{P} g$.

Lemma 2 (quadratic mean convergence implies consistency). If $\|\mathrm{Bias}(T_G)\| \to 0$ and $\mathrm{tr}\,\mathrm{Cov}(T_G) \to 0$, then $T_G \xrightarrow{qm} g$, and consequently $T_G$ is consistent.

Note the converse direction of the intuition fails in an instructive way: an estimator can be biased at every finite $G$ and still be consistent, and it can be consistent for something other than the quantity you wanted. Both phenomena occur below in the blog :D.

Finally, four limit theorems, stated without proof since they are standard (again, can refer Prof. Larry's book). Let $Z_1, Z_2, \dots$ be iid with mean $m$.

• (SLLN) If $\mathbb{E}\|Z_1\| < \infty$, then $\frac{1}{G}\sum_{i=1}^G Z_i \xrightarrow{\text{a.s.}} m$.
• (CLT) If $\mathbb{E}\|Z_1\|^2 < \infty$, then $\sqrt{G}\left( \frac{1}{G}\sum_i Z_i - m \right) \xrightarrow{D} N(0, \mathrm{Cov}(Z_1))$.
• (Slutsky / continuous mapping) If $X_G \xrightarrow{D} X$ and $Y_G \xrightarrow{P} c$ for a constant $c \neq 0$, then $X_G / Y_G \xrightarrow{D} X / c$; and if $T_G \xrightarrow{\text{a.s.}} t$ and $\phi$ is continuous at $t$, then $\phi(T_G) \xrightarrow{\text{a.s.}} \phi(t)$.
• (Delta method) If $\sqrt{G}(T_G - \mu) \xrightarrow{D} N(0, \Sigma)$ and $\phi$ is differentiable at $\mu$, then $\sqrt{G}\left( \phi(T_G) - \phi(\mu) \right) \xrightarrow{D} N\!\left(0, \nabla\phi(\mu)^\top \Sigma \nabla\phi(\mu)\right)$, and under second-order smoothness and integrability conditions $\mathbb{E}[\phi(T_G)] = \phi(\mu) + O(1/G)$, with the $O(1/G)$ term driven by the curvature of $\phi$ and the covariance of $T_G$.

2. The RL problem in LLM post-training

Post-training with a reward model or verifiable rewards is, in its simplest form, a contextual banditContextual bandit. An RL problem with a single decision step. The prompt is the context, the full completion is one action, and there is no further state transition. Treating the completion as one action (rather than each token as an action in an MDP) is what makes the sequence-level analysis in this post exact.. A prompt $x$ is drawn from a distribution $\mathcal{D}$, the policy $\pi_\theta$ (the LLM) samples a full completion $y \sim \pi_\theta(\cdot \mid x)$, and a scalar reward $r(x, y)$ is observed. The objective is

We want to ascend $J$ by stochastic gradient methods, so the fundamental statistical object of the whole enterprise is the gradient $\nabla_\theta J(\theta)$. Everything that follows concerns estimators of this vector.

Proposition 1 (Policy gradient / score function identity). If $\pi_\theta(y \mid x) > 0$ for all $y$ and differentiation under the sum is valid, then

The vector $s_\theta(x, y) = \nabla_\theta \log \pi_\theta(y \mid x)$ is the scoreScore function. The gradient of the log-likelihood, $s_\theta = \nabla_\theta \log \pi_\theta$. Its defining property, $\mathbb{E}[s_\theta] = 0$ under the model, is the engine behind Proposition 2 and also the reason the Fisher information $\mathbb{E}[s_\theta s_\theta^\top]$ is a covariance matrix., and for an autoregressive model it decomposes as a sum of per-token log-probability gradients. This identity is the reason RL on LLMs is possible at all: the reward can be a black box, and only the policy needs to be differentiable.

3. The estimators

3.1 REINFORCE: the plug-in Monte Carlo estimator

Sample one prompt and $G$ completions $y_1, \dots, y_G \overset{\text{iid}}{\sim} \pi_\theta(\cdot \mid x)$, and form

By Proposition 1 and linearity of expectation, $\mathbb{E}[\hat{g}_{\text{MC}}] = \nabla J(\theta)$, so this estimator is exactly unbiased. Its problem is variance. If rewards live in $[0, 1]$, every sampled completion pushes its own log-probability up, and the signal distinguishing good completions from bad ones is buried in the common positive offset. Formally, adding a constant $c$ to all rewards leaves $\nabla J$ unchanged (since $\mathbb{E}[c \, s_\theta] = 0$, as shown next) but inflates the second moment of the estimator by terms of order $c^2 \, \mathbb{E}\|s_\theta\|^2$. An estimator whose variance depends on the arbitrary origin of the reward scale is begging for a control variateControl variate. A variance reduction technique: to estimate $\mathbb{E}[X]$, use $X - c(W - \mathbb{E}[W])$ for a correlated $W$ with known mean. Here $W = b(x) s_\theta$ with known mean zero, so subtracting it is free of bias, and choosing $b$ well makes it strongly negatively correlated with the noise in $r \, s_\theta$..

3.2 Baselines are control variates

Proposition 2 (Baselines do not bias the gradient). For any function $b(x)$ that does not depend on $y$,

So replacing $r$ with the advantage $A = r - b(x)$ preserves unbiasedness while changing the variance. A short calculation identifies the best possible scalar baseline. Writing $s = s_\theta(x, y)$ and minimizing the trace of the covariance of $(r - b) s$ over constants $b$, the objective $\mathbb{E}\|(r - b) s\|^2$ is a quadratic in $b$, and setting its derivative to zero gives

a score-weighted mean reward. In practice everyone uses the unweighted approximation $b(x) \approx \mathbb{E}_{y}[r(x, y)]$, the expected reward of the current policy on that prompt. This quantity is unknown, and the different algorithms are distinguished mostly by how they estimate it.

3.3 RLOO: the leave-one-out baseline

RLOO estimates the baseline for sample $i$ using the other $G - 1$ samples in the group:

Proposition 3 (RLOO is unbiased). $\mathbb{E}[\hat{g}_{\text{RLOO}}] = \nabla J(\theta)$.

The independence step is the entire trick. Had we included $r_i$ in its own baseline, the baseline would correlate with $s_i$ and the argument would fail, which is exactly the door through which GRPO's bias enters.

3.4 GRPO: group normalization

GRPO uses the full group mean and, crucially, divides by the group standard deviation:

The mean subtraction alone is harmless. A one-line algebraic identity shows

so centering by the full-group mean is exactly the RLOO advantage shrunk by the deterministic factor $\frac{G-1}{G}$, and the resulting estimator is unbiased for $\frac{G-1}{G} \nabla J$, which is merely a learning-rate rescaling. The bias comes from $\hat{\sigma}$, for two separate reasons.

First, $\hat{\sigma}$ is a random quantity computed from the same samples, and it is correlated with $r_i$. Since $\mathbb{E}[X / Y] \neq \mathbb{E}[X] / \mathbb{E}[Y]$ in general, the expectation of the ratio does not factor, and for finite $G$ we have $\mathbb{E}[\hat{g}_{\text{GRPO}}] \neq c \, \nabla J(\theta)$ for any constant $c$.

Second, and more interesting, the bias does not vanish into the true gradient even asymptotically. This is the subject of the next section.

3.5 PPO: importance sampling and deliberate bias

PPO reuses samples drawn from a stale policy $\pi_{\text{old}}$. The exact off-policy correction is importance samplingImportance sampling. Unbiased when $q$ covers the support of $p$, but the variance involves $\mathbb{E}_q[(p/q)^2 f^2]$, which is finite only when $p^2 / q$ is integrable. For autoregressive policies the ratio is a product over tokens, so small per-token disagreements compound geometrically.: for $\rho(y) = \pi_\theta(y \mid x) / \pi_{\text{old}}(y \mid x)$,

valid whenever $\pi_{\text{old}}$ has support wherever $\pi_\theta$ does. This estimator is unbiased but its variance is governed by the second moment $\mathbb{E}_{\pi_{\text{old}}}[\rho^2 f^2]$, which involves the chi-square divergence between the two policies and can be astronomically large for sequence-level ratios, since $\rho$ is a product over hundreds of per-token ratios and its distribution is heavy-tailed. PPO responds by optimizing the clipped surrogate

whose gradient is a truncated importance sampling estimator. Truncation caps the variance at the price of bias, since the clipped ratio no longer integrates the tails correctly. This is not an accident of implementation but the entire design philosophy: PPO is the algorithm that chooses a bounded, biased estimator over an unbiased one with potentially infinite variance. Anyone who has seen truncated importance weights or trimmed estimators in a statistics course will recognize the move.

4. Analysis!!

4.1 The bias and the variance

Throughout this section fix the prompt $x$, write $\nabla J_x = \mathbb{E}[r s]$ for the per-prompt gradient, $\mu = \mathbb{E}[r]$, $\sigma^2 = \mathrm{Var}(r)$, and abbreviate $s_i = s_\theta(x, y_i)$. Two lemmas make the informal variance claims of Section 3 exact.

Lemma 3 (variance depends on the reward origin). Shifting all rewards by a constant $c$ leaves the mean of the single-sample estimator $r s$ unchanged but changes its variance by

The variance is therefore a quadratic in the reward origin, minimized exactly at the optimal baseline $b^\star$ of Section 3.2, and unboundedly bad for large offsets. This is the precise version of the statement that raw REINFORCE is at the mercy of the reward scale.

Proposition 4 (exact cost of estimating the baseline). Let $(r, s)$ be one sample and let $B$ be a baseline estimate independent of $(r, s)$ with $\mathbb{E}[B] = \mu$ and $\mathrm{Var}(B) = \sigma^2 / m$, for instance the mean of $m$ independent rewards. Then $(r - B)s$ is unbiased for $\nabla J_x$ and

Applied to RLOO, where the baseline for sample $i$ is the mean of the $m = G - 1$ other rewards, the price of not knowing $\mu$ is exactly $\frac{\sigma^2}{G - 1}\mathbb{E}\|s\|^2$ per term, which vanishes as the group grows. The ledger, then. REINFORCE is unbiased with variance hostage to the reward origin (Lemma 3). RLOO is unbiased with the origin term removed and an $O\!\left(\frac{1}{G-1}\right)$ surcharge for estimating the baseline (Proposition 4). GRPO is biased at every finite $G$ through the random denominator, quantified below. The clipped PPO gradient is biased even as $G \to \infty$, by construction, in exchange for bounded variance under off-policy reuse.

4.2 Consistency and asymptotics

ConsistencyGRPO at fixed $G$ is a biased estimator that is consistent for a different (reweighted) target, which is a useful reminder that consistency is a statement about the limit, and one must always ask what the limit actually is. asks what the estimators converge to as $G \to \infty$ with the prompt fixed. Define the centered numerator that both RLOO and GRPO share,

and recall from the identity in Section 3.4 that $\hat{g}_{\text{RLOO}} = \frac{G}{G - 1} N_G$ and $\hat{g}_{\text{GRPO}} = N_G / \hat{\sigma}$.

Proposition 5 (consistency and rate for REINFORCE and RLOO). Assume $\mathbb{E}\|r s\|^2 < \infty$ and $\mathbb{E}\|s\|^2 < \infty$. Then $\hat{g}_{\text{MC}} \xrightarrow{\text{a.s.}} \nabla J_x$ and $\hat{g}_{\text{RLOO}} \xrightarrow{\text{a.s.}} \nabla J_x$, and $\sqrt{G}\left( \hat{g}_{\text{MC}} - \nabla J_x \right) \xrightarrow{D} N(0, \mathrm{Cov}(r s))$.

If in addition $|r| \leq R$ and the score coordinates are bounded by $S$, Hoeffding's inequality applied per coordinate of $\hat{g}_{\text{MC}}$ gives the non-asymptotic tail bound $\mathbb{P}\left( |\hat{g}_j - (\nabla J_x)_j| \geq t \right) \leq 2 \exp\!\left( - \frac{G t^2}{2 R^2 S^2} \right)$, so the root-$G$ rate holds at every finite sample size and not merely in the limit.

GRPO is where the asymptotics become genuinely interesting, and where the continuous mapping theoremSlutsky's theorem and the continuous mapping theorem together are the tools that let us replace the random $\hat{\sigma}$ with its limit $\sigma(x)$ when computing what GRPO converges to, as in the proof of Proposition 6. earns its keep.

Proposition 6 (GRPO is consistent for a reweighted gradient). Assume $\mathbb{E}\|r s\| < \infty$, $\mathbb{E}[r^2] < \infty$, and $\sigma > 0$. Then

So GRPO is consistent, but not for $\nabla J_x$. It is consistent for the gradient of each prompt's objective reweighted by $1 / \sigma(x)$, and averaged over prompts it ascends $\tilde{J}(\theta) = \mathbb{E}_x \left[ J_x(\theta) / \sigma_x \right]$ in a first-order sense. This upweights prompts where the policy's rewards are nearly deterministic and downweights high-variance prompts. Whether this implicit reweighting is a feature or a bug is a modeling question rather than a statistical one, but the statistics forces us to notice that it exists.

Proposition 7 (GRPO's finite-sample bias is $O(1/G)$). Assume rewards are bounded, scores are bounded, and $\hat{\sigma}$ is bounded below by some $\epsilon > 0$, as it is in implementations that add $\epsilon$ to the denominator. Then

The $O(1/G)$ scaling comes from a second-order expansion of the reciprocal standard deviation.A second-order expansion of the map $v \mapsto v^{-1/2}$ (the delta method) is what produces the $O(1/G)$ bias estimate for GRPO's normalized advantage, since the first-order term has mean zero and the curvature of $(u, v) \mapsto u / v$ contributes at order $1/G$.

This is why published ablations find GRPO and its debiased variants nearly indistinguishable at common group sizes like $G = 16$ or $64$, and why variants such as Dr. GRPO simply delete the $\hat{\sigma}$ to restore an unbiased target. The bias is real but it is a $1/G$ effect stacked on top of a $1/\sqrt{G}$ noise floor, so at moderate group sizes the noise dominates the bias by an order of magnitude.

For clipped PPO no amount of sampling helps. The estimator converges by the law of large numbers to the gradient of the clipped surrogate, which differs from $\nabla J$ whenever the clip is active with positive probability. The bias is a fixed feature of the estimand, not a finite-sample artifact, and the algorithm is best understood as consistently estimating the gradient of a deliberately different, trust-region-flavored objective.

4.3 Why any of this matters for training

Stochastic gradient ascent with step sizes $\eta_t$ satisfying the Robbins-Monro conditions $\sum_t \eta_t = \infty$ and $\sum_t \eta_t^2 < \infty$ converges to a stationary point when the gradient estimator is unbiased with bounded variance. With a persistent bias, the guarantee degrades to a neighborhood whose radius scales with the bias, and this can be proved in a few lines with the standard smoothness toolkit.

Proposition 8 (biased SGD reaches a bias-sized neighborhood). Suppose $J$ is $L$-smooth and bounded above by $J^\star$, and run $\theta_{t+1} = \theta_t + \eta \, \hat{g}_t$ with a constant step $\eta \leq \frac{1}{4L}$, where $\hat{g}_t = \nabla J(\theta_t) + b_t + \xi_t$ with $\mathbb{E}[\xi_t \mid \theta_t] = 0$, $\mathbb{E}[\|\xi_t\|^2 \mid \theta_t] \leq v$, and $\|b_t\| \leq \beta$. Then

The first term vanishes as $T \to \infty$, the last is controlled by shrinking the step, but $3\beta^2$ is a floor that no amount of training removes. This is the precise sense in which the taxonomy above has teeth. RLOO's unbiasedness buys clean convergence guarantees. GRPO's $O(1/G)$ bias buys a neighborhood that shrinks with group size, which is cheap. PPO's non-vanishing bias means the fixed points themselves move, which is tolerated because the surrogate's fixed points are close to the original ones when updates stay inside the trust region. Variance, meanwhile, controls the constants and the usable learning rate everywhere. The whole field is a walking bias-variance tradeoff, and the vocabulary from a first estimation theory course is exactly the right one for reading it.

5. ....umm all that's fine, but what happens in practice?

Everything so far assumed we can sample from $\pi_\theta$ and evaluate $\nabla \log \pi_\theta$ exactly. In a production post-training system neither assumption is quite true, and the failure modes are best understood in the vocabulary of the previous sections.

5.1 The sampler is not the trainer

Real pipelines split into two engines. Rollouts are generated by an inference engineInference engine. A serving system such as vLLM or SGLang that generates tokens at high throughput using techniques like paged KV caching, continuous batching, prefix caching, and fused or quantized kernels. RL pipelines pair one of these with a separate training framework because generation is memory-bandwidth bound while gradient computation is compute bound, and the two workloads want different kernels. The statistical price of this efficiency is that the generating distribution and the differentiated distribution are computed by different code paths. built for throughput, with paged KV caches, fused kernels, continuous batching, and often reduced precision, while log-probabilities and gradients are computed by a training engine with different kernels, different reduction orders, and different precision policies. Even with bit-identical weights, the two engines realize two different conditional distributions, $\pi_{\text{sample}} \neq \pi_{\text{train}}$, because floating point arithmetic is not associativeFloating point non-associativity. In finite precision, $(a + b) + c$ need not equal $a + (b + c)$, so the value of a large reduction like a matrix multiply or a softmax normalizer depends on the order in which terms are accumulated. Different kernels, different batch sizes, and different parallelism layouts induce different orders, which is why two mathematically identical implementations of the same network produce slightly different probabilities, and why bitwise reproducibility requires deliberately batch-invariant kernels. and per-token probabilities disagree at the level of $10^{-3}$ or so in BF16. The statistical consequence is that supposedly on-policy training is silently off-policy: samples come from $q = \pi_{\text{sample}}$ but the score is evaluated under $p = \pi_{\text{train}}$, so the plug-in estimator targets $\mathbb{E}_q[r \, \nabla \log p]$ rather than $\nabla J$. A per-token discrepancy of $10^{-3}$ sounds ignorable until you recall that the sequence-level ratio is a product over hundreds or thousands of tokens, so log discrepancies add and the ratio drifts exponentially in length. The remedies are exactly the menu from Section 3.5. One can correct with full sequence-level importance weights, which restores unbiasedness at the price of heavy-tailed variance. One can correct with truncated or per-token importance weights, accepting a small bias for bounded variance, which is the choice most open frameworks have converged on. Or one can attack the bias at its source by aligning numerics, for instance computing sampler log-probabilities in FP32 and using batch-invariant deterministic kernels, which costs throughput but shrinks $\chi^2(p \, \| \, q)$ toward zero so that no correction is needed. Which point on this frontier is optimal is a systems question, but the frontier itself is the bias-variance tradeoff of Section 4.

5.2 Precision, determinism, and what is bias versus what is variance

Numerical effects sort cleanly into the two columns of the ledger. Systematic differences are bias: a sampler that always computes softmax in BF16 while the trainer uses FP32 shifts the sampling distribution in a fixed direction, quantized KV caches in serving perturb every rollout the same way, and gradient clipping truncates the same tail on every step. Nondeterministic differences are variance: atomics and reduction-order nondeterminism inject approximately mean-zero noise into log-probabilities and gradients, and continuous batching makes a sequence's logits depend on which requests it happened to be batched with, which acts like extra sampling noise when batch composition is random. The distinction matters operationally because Proposition 8 says variance can be averaged away with larger batches and smaller steps while bias cannot. This also explains a phenomenon that surprises people at scale: as $G$ and the batch size grow, the $1/\sqrt{G}$ noise floor drops until the residual systematic terms, sampler mismatch, clipping bias, and reward model artifacts, become the binding constraint. Small-scale ablations live in the variance-dominated regime and large-scale runs live in the bias-dominated regime, so conclusions do not transfer for a reason you can state as an inequality between $\beta^2$ and $2 L \eta v$.

5.3 Serving economics shape the estimator

The estimator choices also interact with inference costs in ways that are easy to miss. Group-based methods like GRPO and RLOO are popular partly because $G$ completions of the same prompt share a prefill, so prefix caching makes within-prompt samples far cheaper than samples of fresh prompts, which is a variance-reduction subsidy paid by the serving stack. Asynchronous actor-learner designs raise throughput by letting rollouts lag the learner by a few versions, but staleness widens the gap between behavior and target policies, inflates the chi-square divergence that controls importance weight variance, activates the PPO clip more often, and therefore converts throughput into bias. Length is a lurking variable throughout: sequence-level ratios and sequence-level rewards both scale badly with completion length, and normalization choices that look innocuous, such as dividing token losses by the completion length, reweight the objective across lengths in the same way GRPO's $\hat{\sigma}$ reweights it across prompts.

5.4 Some open questions.....and discussion :)

I was recently interviewing for a couple of frontier labs, and during my prep (and during the interviews), I came across so many nuances when theory is actually applied to practice. While inspired from the interviews, the following questions' discussion is strictly my own POV and could be wrong (please do reach out with any concerns, such discussions are always welcome).

Question. Your RL run generates rollouts with an inference engine and trains in BF16. Training is stable for a thousand steps and then collapses. A colleague suggests the run is secretly off-policy. What diagnostic would you look at, and what are the possible fixes with their tradeoffs?

Question. You double the GRPO group size from $G = 32$ to $G = 64$. What improves, by how much, and when does this stop being worth it?

Question. The unbiased off-policy correction is the sequence-level importance ratio, yet essentially every framework applies ratios and clipping per token. Defend the practice, then attack it.

Question. You must estimate and penalize the KL divergence to a reference policy. Compare the estimators $k_1 = -\log \rho$ and $k_3 = (\rho - 1) - \log \rho$ with $\rho = \pi_{\text{ref}} / \pi_\theta$ on samples from $\pi_\theta$, and explain what changes when the penalty enters the reward versus the loss.

Question. Two runs with identical seeds, data order, and hyperparameters diverge after ten thousand steps. Is nondeterminism a bias problem or a variance problem, and does it threaten the consistency arguments of Section 4?