Introducing Background Temperature to Characterise Hidden Randomness in Large Language Models

arXiv:2604.22411v1 Announce Type: new
Abstract: Even when decoding with temperature $T=0$, large language models (LLMs) can produce divergent outputs for identical inputs. Recent work by Thinking Machines Lab highlights implementation-level sources of nondeterminism, including batch-size variation, kernel non-invariance, and floating-point non-associativity. In this short note we formalize this behavior by introducing the notion of emphbackground temperature $T_mathrmbg$, the effective temperature induced by an implementation-dependent perturbation process observed even when nominal $T=0$. We provide clean definitions, show how $T_mathrmbg$ relates to a stochastic perturbation governed by the inference environment $I$, and propose an empirical protocol to estimate $T_bg$ via the equivalent temperature $T_n(I)$ of an ideal reference system. We conclude with a set of pilot experiments run on a representative pool from the major LLM providers that demonstrate the idea and outline implications for reproducibility, evaluation, and deployment.