Responsible machine learning in child welfare and digital mental health

Abstract

Machine learning-based technologies are increasingly being used to assist care work in high-stakes domains, from provisioning resources for poor families to providing mental health support for people experiencing distress. These technologies have been introduced in hopes that they improve care and decision quality. Depending on how they are designed and used, these technologies may also perpetuate harms like racial discrimination or carcerality. In order to understand how machine learning technologies can harm or help, it is necessary to understand the perspectives of people who use these technologies or are impacted by them. Yet, in many high-stakes domains, the perspectives of impacted people ---especially those who are marginalized or do not have the ability to directly influence the design of these technologies--- remain overlooked. This dissertation presents case studies of evaluating and designing machine learning technologies in child welfare and digital mental health through both quantitative and qualitative methods with people who may be impacted by machine learning technologies. Within child welfare, I explore how existing algorithmic decision-making tools exacerbate harms. First, I evaluate a particular algorithm used in the child welfare system to understand how workers use it to reduce or exacerbate racial biases. Second, I engage impacted people like parents and workers to understand how these algorithmic technologies replicate further systemic harms like carcerality. I then explore how we might design different technologies to benefit those most marginalized by the child welfare system. Within digital mental health, I continue to explore how AI-based technologies might be designed or deployed responsibly in this space, if at all. I use participatory design to understand how digital mental health support providers approach suicide prevention online and whether they think machine learning technologies could benefit them while preventing harms to support seekers. Finally, based on suggestions from mental health support providers, I design and evaluate conversational agents that simulate people in distress to help train new support providers. This work aims to showcase ways to understand how machine learning technologies exacerbate systemic harms and how we might design them better.