Talia Gillis1, Bryce McLaughlin2, Jann Spiess1
1 Columbia University, New York, United States
2 Stanford University, California, United States
When machine-learning algorithms are deployed in high-stakes decisions, we want to ensure that their deployment leads to fair and equitable outcomes. This concern has motivated a fast-growing literature that focuses on diagnosing and addressing disparities in machine predictions. However, many machine predictions are deployed to assist in decisions where a human decision-maker retains the ultimate decision authority. In this article, we therefore consider how properties of machine predictions affect the resulting human decisions. We show in a formal model that the inclusion of a biased human decision-maker can revert common relationships between the structure of the algorithm and the qualities of resulting decisions. Specifically, we document that excluding information about protected groups from the prediction may fail to reduce, and may even increase, ultimate disparities. While our concrete results rely on specific assumptions about the data, algorithm, and decision-maker, they show more broadly that any study of critical properties of complex decision systems, such as the fairness of machineassisted human decisions, should go beyond focusing on the underlying algorithmic predictions in isolation.
We thank Asa Palley, Stefan Wager, Daniel Kipnis, Sendhil Mullainathan, Ashesh Rambachan, Larry Wein, as well as conference audiences at the 2021 INFORMS annual meeting, the 2022 FAccT conference, the 2022 MSOM conference and the 2022 ALEA annual meeting for helpful discussions, comments, and suggestions.