Context Channel Capacity: An Information-Theoretic Framework for Understanding Catastrophic Forgetting

arXiv:2603.07415v1 Announce Type: cross
Abstract: Catastrophic forgetting remains a central challenge in continual learning (CL), yet lacks a unified information-theoretic explanation for why some architectures forget catastrophically while others do not. We introduce emphContext Channel Capacity ($C_mathrmctx$), the mutual information between a CL architecture’s context signal and its generated parameters, and prove that zero forgetting requires $C_mathrmctx geq H(T)$, where $H(T)$ is the task identity entropy. We establish an emphImpossibility Triangle — zero forgetting, online learning, and finite parameters cannot be simultaneously satisfied by sequential state-based learners — and show that conditional regeneration architectures (HyperNetworks) bypass this triangle by redefining parameters as function values rather than states.
We validate this framework across 8 CL methods on Split-MNIST (1,130+ experiments over 86 days, 4 seeds each), showing that $C_mathrmctx$ perfectly predicts forgetting behavior: methods with $C_mathrmctx = 0$ (NaiveSGD, EWC, SI, LwF, CFlow) exhibit catastrophic forgetting (6–97%), while methods with $C_mathrmctx approx 1$ (HyperNetwork) achieve zero forgetting (98.8% ACC). We further propose emphWrong-Context Probing (P5), a practical diagnostic protocol for measuring $C_mathrmctx$, and extend the framework to CIFAR-10 via a novel emphGradient Context Encoder that closes the oracle gap from 23.3pp to 0.7pp. A systematic taxonomy of 15+ closed research directions — including the Hebbian null result (frozen random features outperform learned features), CFlow’s $theta_0$-memorizer phenomenon, and the $S_N$ symmetry barrier to column specialization — provides the community with precisely diagnosed negative results. Our central design principle: empharchitecture over algorithm — the context pathway must be structurally unbypassable.