Exact Unlearning in Reinforcement Learning

arXiv:2606.04182v1 Announce Type: cross
Abstract: We formulate the problem of emphexact unlearning in reinforcement learning, where the goal is to design an efficient framework that enables the removal of any user’s data upon deletion request, i.e., the online learner’s output after unlearning is emphindistinguishable from what would have been produced had the deleted user never interacted with the learner. For any $rho >0$, we show that there exists a reinforcement learning (RL) algorithm that is $rho$-TV-stable and supports an exact unlearning procedure whose expected computational cost is only a $rho sqrtln T$ fraction of the computational cost of retraining from scratch. We construct such a $rho$-TV-stable RL algorithm for tabular Markov decision processes (MDPs), which achieves a regret bound of $mathcalO(H^2 sqrtSAT + H^3 S^2 A + H^2.5 S^2 A/rho)$, where $S, A, H$, and $T$ denote the number of states, the number of actions, the episode horizon, and the number of episodes, respectively. We also establish a lower bound of $Omega(Hsqrt!SAT! +! SAH/rho)$ for $rho$-TV-stable RL algorithms, showing that our algorithm is nearly minimax optimal.