How do LLMs Compute Verbal Confidence

arXiv:2603.17839v2 Announce Type: replace-cross
Abstract: Verbal confidence — prompting LLMs to state their confidence as a number or category — is widely used to extract uncertainty estimates from black-box models. However, how LLMs internally generate such scores remains unknown. We address two questions: first, when confidence is computed – just-in-time when requested, or automatically during answer generation and cached for later retrieval; and second, what verbal confidence represents – token log-probabilities, or a richer evaluation of answer quality? Focusing on Gemma 3 27B and Qwen 2.5 7B, we provide convergent evidence for cached retrieval. Activation steering, patching, noising, and swap experiments reveal that confidence representations emerge at answer-adjacent positions before appearing at the verbalization site. Attention blocking pinpoints the information flow: confidence is gathered from answer tokens, cached at the first post-answer position, then retrieved for output. Critically, linear probing and variance partitioning reveal that these cached representations explain substantial variance in verbal confidence beyond token log-probabilities, suggesting a richer answer-quality evaluation rather than a simple fluency readout. These findings demonstrate that verbal confidence reflects automatic, sophisticated self-evaluation — not post-hoc reconstruction — with implications for understanding metacognition in LLMs and improving calibration.