Liat Antwarg Friedman & Maya Dagan & Maya Makov

Machine learning models in healthcare operate under constraints that differ fundamentally from those in many other data science domains. Outcomes are rare, data are highly imbalanced, and downstream resources, such as diagnostic tests or clinical follow-up are limited. In these settings, overall discrimination metrics are often insufficient. What matters most is performance at the very top of the ranked list, where only a small fraction of patients can be acted upon.
The first part of this talk includes interesting points in training and evaluating predictive models for such resource-constrained healthcare settings, and why head-of-list metrics better reflect real-world value than global performance measures.
However, strong head-of-list performance alone may not be enough. The second part of the talk will include how this became clear during deployment within Clalit’s Proactive-Preventive Intervention (C-Pi) platform, where patients are prioritized for proactive outreach. Although prediction models performed well on head-of-list metrics, clinicians often disagreed with the top-ranked patients – particularly when very elderly, high-risk but low-actionability patients dominated the list. This exposed a deeper challenge: prioritization requires modeling clinical trade-offs beyond risk alone. To address this gap, we developed a learning-based ranking approach that explicitly models how clinicians prioritize patients in practice. We will present this method and discuss what it reveals about the gap between risk prediction and real world clinical prioritization.