ArXiv Preprint
Across domains such as medicine, employment, and criminal justice, predictive
models often target labels that imperfectly reflect the outcomes of interest to
experts and policymakers. For example, clinical risk assessments deployed to
inform physician decision-making often predict measures of healthcare
utilization (e.g., costs, hospitalization) as a proxy for patient medical need.
These proxies can be subject to outcome measurement error when they
systematically differ from the target outcome they are intended to measure.
However, prior modeling efforts to characterize and mitigate outcome
measurement error overlook the fact that the decision being informed by a model
often serves as a risk-mitigating intervention that impacts the target outcome
of interest and its recorded proxy. Thus, in these settings, addressing
measurement error requires counterfactual modeling of treatment effects on
outcomes. In this work, we study intersectional threats to model reliability
introduced by outcome measurement error, treatment effects, and selection bias
from historical decision-making policies. We develop an unbiased risk
minimization method which, given knowledge of proxy measurement error
properties, corrects for the combined effects of these challenges. We also
develop a method for estimating treatment-dependent measurement error
parameters when these are unknown in advance. We demonstrate the utility of our
approach theoretically and via experiments on real-world data from randomized
controlled trials conducted in healthcare and employment domains. As
importantly, we demonstrate that models correcting for outcome measurement
error or treatment effects alone suffer from considerable reliability
limitations. Our work underscores the importance of considering intersectional
threats to model validity during the design and evaluation of predictive models
for decision support.
Luke Guerdan, Amanda Coston, Kenneth Holstein, Zhiwei Steven Wu
2023-02-22
