$pi$-Attention: Periodic Sparse Transformers for Efficient Long-Context Modeling

arXiv:2511.10696v2 Announce Type: replace-cross
Abstract: Transformers have revolutionized natural language processing, but their quadratic complexity with respect to sequence length remains a fundamental bottleneck for long-range modeling. While sparse attention mechanisms like RingAttention reduce computational costs by restricting attention to local neighborhoods, they suffer from limited receptive fields and lack of adaptability. We present PiAttention, a periodic sparse Transformer that factorizes attention into ring-local neighborhoods, deterministic $pi$-stride skips, and an adaptive fusion gate. The periodic structure provides predictable coverage of distant tokens, while the sparse footprint keeps the per-layer complexity linear in context length. We prove that PiAttention achieves $mathcalO(kL + pi log L)$ receptive field growth compared to $mathcalO(kL)$ for RingAttention, where $k$ is the local window size, $pi$ is the skip period, and $L$ is the sequence length. Extensive experiments on language modeling, retrieval, and vision-language tasks demonstrate that PiAttention matches or surpasses dense attention quality with 8.3% lower perplexity than RingAttention while using 50% fewer GPUs for the same context length. Our detailed ablations and visualizations reveal the importance of periodic skips, adaptive fusion, and head-level sparsity coordination for efficient long-context modeling.