Learning to Defer with Scoring Functions

Machine learning models in high-stakes applications often collaborate with human experts, requiring intelligent deferral mechanisms. While traditional rejection learning uses model uncertainty to decide deferrals, it ignores human error and lacks flexibility in coverage control. Learning-to-defer approaches address this by jointly optimizing a classifier and a rejector, using loss functions that account for both model and human performance. However, existing methods either disregard human resource constraints or require costly retraining when coverage needs change. We explore a scoring-based approach to learning to defer that addresses these limitations. Building on prior heuristic methods, we introduce a novel metric quantifying how effectively a deferral system leverages both model and human expertise across the feature space. We then derive the theoretically optimal deferral rule and develop a practical scoring function approximation that enables post-hoc coverage adjustment without retraining. Our method trains a scoring function to rank samples by their expected delegation benefit, which can be calibrated to meet dynamic coverage constraints. Experiments show this approach achieves superior accuracy-workload trade-offs compared to existing methods, providing both theoretical grounding and practical flexibility for human-AI collaboration.