SUMMARY - Community Involvement in AI Design
A healthcare algorithm is developed by engineers and data scientists in a technology company, tested by clinicians at academic medical centers, and deployed across hospitals serving diverse populations whose health patterns, care-seeking behaviors, and life circumstances differ dramatically from those who designed and validated the system. A predictive policing tool is purchased by a city government without consulting the neighborhoods where it will direct police attention, communities with long histories of over-policing who could have predicted the harms that later materialized.