When to Stop Reusing: Dynamic Gradient Gating for Sample-Efficient RLVR

arXiv:2605.19425v1 Announce Type: cross
Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant paradigm for advanced reasoning in Large Language Models (LLMs), but rollout samples are expensive to obtain, making sample efficiency a critical bottleneck. A natural remedy is to reuse each rollout batch for multiple gradient updates, a standard practice in classical RL. Yet in RLVR, this amplifies policy shift, leading to severe performance degradation. Detecting the onset of degradation early enough to stop reuse remains an open and challenging problem. We close this gap by identifying the textitDisproportionate Weight Divergence (DWD) phenomenon: performance degradation is synchronized with a sharp surge in the textttlm_head weight change, while intermediate layers remain stable. Empirically, we verify that DWD emerges consistently across diverse LLMs and tasks. Theoretically, we prove that (i) harmful gradients concentrate at the textttlm_head while intermediate layers are structurally attenuated, and (ii) the textttlm_head gradient norm lower-bounds the policy divergence. These results establish the textttlm_head gradient norm as a principled, real-time signal of catastrophic policy shift. Guided by this insight, we propose textitDynamic Gradient Gating (DGG), a lightweight intervention that monitors the textttlm_head gradient norm in real time and intercepts harmful gradients before they corrupt the optimizer. DGG consistently matches or exceeds the standard single-use baseline, achieving up to $2.93times$ sample efficiency and $2.14times$ wall-clock speedup across math, ALFWorld, WebShop, and search-augmented QA tasks.