PathRWKV: Enhancing Whole Slide Image Inference with Asymmetric Recurrent Modeling

arXiv:2503.03199v4 Announce Type: replace-cross
Abstract: Whole Slide Imaging (WSI) has become a gold standard in cancer diagnosis, inspecting multi-scale information from cellular to tissue levels. Processing an entire WSI directly is infeasible due to GPU memory constraints; thus, Multiple Instance Learning (MIL) has emerged as the standard solution by partitioning WSIs into tiles. While recent two-stage MIL frameworks partially achieve memory efficiency by decoupling tile-level extraction from slide-level modeling, they still face four limitations: (1) the conflict between training throughput and inference memory efficiency, (2) the high susceptibility to overfitting on small-scale WSI datasets with sparse supervision, (3) the disruption of spatial structural integrity during sampling-based training, and (4) the inadequate modeling of multi-scale feature interactions within long sequences. We therefore introduce PathRWKV, a novel State Space Model designed for efficient and robust WSI analysis. To resolve the computational trade-off, we propose an asymmetric structure utilizing max pooling aggregation, enabling parallelized training for high throughput and recurrent inference with constant (O(1)) memory complexity. To mitigate overfitting, we employ random sampling to enhance data diversity, with a multi-task learning module to regularize feature learning on limited data. To restore spatial context, we introduce 2D sinusoidal position encoding to perceive the relative locations of tissue tiles. To capture comprehensive representations, we integrate TimeMix and ChannelMix modules, enabling dynamic multi-scale feature modeling across temporal and spatial dimensions. Experiments on 29,073 WSIs across 11 datasets demonstrate that PathRWKV outperforms 11 state-of-the-art methods on 10 datasets, establishing it as a scalable and solution with application potential.