Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignment

arXiv:2601.17329v1 Announce Type: cross
Abstract: Preference-based alignment like Reinforcement Learning from Human Feedback (RLHF) learns from pairwise preferences, yet the labels are often noisy and inconsistent. Existing uncertainty-aware approaches weight preferences, but ignore a more fundamental factor: the reliability of the emphanswers being compared. To address the problem, we propose Conformal Feedback Alignment (CFA), a framework that grounds preference weighting in the statistical guarantees of Conformal Prediction (CP). CFA quantifies answer-level reliability by constructing conformal prediction sets with controllable coverage and aggregates these reliabilities into principled weights for both DPO- and PPO-style training. Experiments across different datasets show that CFA improves alignment robustness and data efficiency, highlighting that modeling emphanswer-side uncertainty complements preference-level weighting and yields more robust, data-efficient alignment. Codes are provided here.