Theoretical Foundations of delta-margin Majority Voting

arXiv:2111.06390v4 Announce Type: replace-cross
Abstract: In high-stakes ML applications such as fraud detection, medical diagnostics, and content moderation, practitioners rely on consensus-based approaches to control prediction quality. A particularly valuable technique — deltadelta delta-margin majority voting — collects votes sequentially until one label exceeds alternatives by a threshold deltadelta delta, offering stronger confidence than simple majority voting. Despite widespread adoption, this approach has lacked rigorous theoretical foundations, leaving practitioners reliant on heuristics for key metrics like expected accuracy and cost.
This paper establishes a comprehensive theoretical framework for deltadelta delta-margin majority voting by formulating it as an absorbing Markov chain and leveraging Gambler’s Ruin theory. Our contributions form a practical emphdesign calculus for deltadelta delta-margin voting: (1)~Closed-form expressions for consensus accuracy, expected voting duration, variance, and the stopping-time PMF, enabling model-based design rather than trial-and-error. (2)~A Bayesian extension handling uncertainty in worker accuracy, supporting real-time monitoring of expected quality and cost as votes arrive, with single-Beta and mixture-of-Betas priors. (3)~Cost-calibration methods for achieving equivalent quality across worker pools with different accuracies and for setting payment rates accordingly.
We validate our predictions on two real-world datasets, demonstrating close agreement between theory and observed outcomes. The framework gives practitioners a rigorous toolkit for designing deltadelta delta-margin voting processes, replacing ad-hoc experimentation with model-based design where quality control and cost transparency are essential.