Hi, I'm Manan. I often find myself taking deep dives on the most random topics on the web, so I created the blog to dump my mind- map here 🫴
Every algorithm in the alphabet soup of PPO, GRPO, and RLOO is, at its core, a Monte Carlo estimator of a policy gradient, and the differences between them are almost entirely differences in how they trade bias for variance. Starting from a first course in estimation theory, this post makes that precise: which estimators are unbiased, which are consistent and for what, and how the finite-sample bias of GRPO's group normalization, the deliberate bias of PPO's clip, and the systems stack's sampler/trainer mismatch all fall out of the same ledger.
Why the most interesting number in a spectrum is almost never the first one. The gap between the first and second eigenvalue quietly controls an astonishing range of systems — random walks mix because of it, clusters exist because of it, quantum phases of matter are classified by it, and the depth pathologies of GNNs trace back to it. One thread, with the actual mathematics: Cheeger's inequality, mixing times, spectral clustering, and the undecidability of the spectral gap.