Tail-Aware Information-Theoretic Generalization for RLHF and SGLD

arXiv:2604.10727v1 Announce Type: cross
Abstract: Classical information-theoretic generalization bounds typically control the generalization gap through KL-based mutual information and therefore rely on boundedness or sub-Gaussian tails via the moment generating function (MGF). In many modern pipelines, such as robust learning, RLHF, and stochastic optimization, losses and rewards can be heavy-tailed, and MGFs may not exist, rendering KL-based tools ineffective. We develop a tail-dependent information-theoretic framework for sub-Weibull data, where the tail parameter $theta$ controls the tail heaviness: $theta=2$ corresponds to sub-Gaussian, $theta=1$ to sub-exponential, and $0<theta<1$ to genuinely heavy tails. Our key technical ingredient is a decorrelation lemma that bounds change-of-measure expectations using a shifted-log $f_theta$-divergence, which admits explicit comparisons to R’enyi divergence without MGF arguments. On the empirical-process side, we establish sharp maximal inequalities and a Dudley-type chaining bound for sub-Weibull processes with tail index $theta$, with complexity scaling as $log^1/theta$ and entropy$^1/theta$. These tools yield expected and high-probability PAC-Bayes generalization bounds, as well as an information-theoretic chaining inequality based on multiscale R’enyi mutual information. We illustrate the consequences in R’enyi-regularized RLHF under heavy-tailed rewards and in stochastic gradient Langevin dynamics with heavy-tailed gradient noise.