The Variance Brain Foundation Models Forgot: Third-Order Statistics Predict Cognition Where Billion-Parameter Models Fail

arXiv:2606.04010v1 Announce Type: new
Abstract: Brain foundation models (BFMs) are self-supervised Transformers pretrained on fMRI data. We posit that these models should capture each subject’s cognitive performance from their fMRI signal. Yet across three state-of-the-art BFMs and every readout we test, they predict cognition worse than a linear regression from the $sim$80K parameters of the functional connectivity matrix (FC). The gap widens with scale: BrainLM’s 650M model predicts cognition worse than its 111M. We attribute this to a textbfvariance allocation problem: BFM pretraining captures the variance components that dominate fMRI but not the higher-order structure that predicts cognition. Our per-cumulant analysis of the reconstructed signal shows that the second-order covariance is partially preserved, while the third-order co-skewness tensor is largely destroyed. To recover what BFMs lose, we design a linear pipeline that projects the fMRI signal into the subspace that best preserves its co-skewness and computes FC there. This textbfexceeds raw FC and every pretrained BFM on every dataset and parcellation we test, outperforming prior state-of-the-art under controlled evaluation textbfwith no pretraining and no GPU. We textbfrecover the raw-FC ceiling on BrainLM’s forward pass by finetuning with a loss targeted at this same subspace. This shows that the bottleneck is the pretraining objective, not the architecture or the model size.