Stateful Reasoning via Insight Replay

arXiv:2605.14457v2 Announce Type: replace
Abstract: Chain-of-Thought (CoT) reasoning has become a foundation for eliciting multi-step reasoning in large language models, but recent studies show that its benefits do not scale monotonically with chain length: while longer CoT generally enables a model to tackle harder problems, on a given problem, accuracy typically increases with CoT length up to a point, after which it declines. We identify a major cause of this phenomenon: as the CoT grows, the model’s attention to critical insights produced earlier in the trace gradually weakens, making those insights progressively less accessible when they are most needed. Therefore, we propose textbfInsightReplay, a stateful reasoning approach in which the model periodically extracts critical insights from its reasoning trace and replays them near the active generation frontier, keeping them accessible as the reasoning scales. Extensive experiments on a $mathbf2!times!mathbf3!times!mathbf4$ benchmark grid, covering model scales $\text8B, text30B$, model families $\textQwen3.5, textDeepSeek-R1-Distill-Qwen, textGemma-4$, and reasoning benchmarks $\textAIME, textHMMT, textGPQA Diamond, textLiveCodeBench v5$, show that 3-round InsightReplay yields accuracy gains across textbfall 24 settings, with an averaged improvement of $mathbf+1.65$ points over standard CoT, and a largest single-setting gain of $mathbf+9.2$ points on R1-Distill-32B’s LiveCodeBench v5 subset. Our results suggest that the effectiveness of test-time scaling depends not only on how much a model reasons, but also on whether critical intermediate insights remain accessible throughout long reasoning trajectories.