Structural intervention is required to mitigate the racially discriminatory impacts of emerging digital technologies including AI

July 15, 2020

[This work was led by UN Special Rapporteur on contemporary forms of racism, racial discrimination, xenophobia and related intolerance and Faculty Director of the Promise Institute for Human Rights at UCLA School of Law Professor E. Tendayi Achiume.]

As widespread recent protests have highlighted, racial inequality remains an urgent and devastating issue around the world, and this is as true in the context of technology as it is everywhere else. In fact, it may be more so, as algorithmic technologies based on big data are deployed at previously unimaginable scale, reproducing the discriminatory systems that build and govern them.

The undersigned organizations welcome the publication of the report “Racial discrimination and emerging digital technologies: a human rights analysis,” by Special Rapporteur on contemporary forms of racism, racial discrimination, xenophobia and related intolerance, E. Tendayi Achiume, and wish to underscore the importance and timeliness of a number of the recommendations made therein:

1. Technologies that have had or will have significant racially discriminatory impacts should be banned outright.
   While incremental regulatory approaches may be appropriate in some contexts, where a technology is demonstrably likely to cause racially discriminatory harm, it should not be deployed until that harm can be prevented. Moreover, certain technologies may always have disparate racial impacts, no matter how much their accuracy can be improved. In the present moment, racially discriminatory technologies include facial and affect recognition technology and so-called predictive analytics. We support Special Rapporteur Achiume’s call for mandatory human rights impact assessments as a prerequisite for the adoption of new technologies. We also believe that where such assessments reveal that a technology has a high likelihood of deleterious racially disparate impacts, states should prevent its use through a ban or moratorium. We join the Special Rapporteur in welcoming recent municipal bans, for example, on the use of facial recognition technology, and encourage national governments to adopt similar policies.  Correspondingly, we reiterate our support for states’ imposition of an immediate moratorium on the trade and use of privately developed surveillance tools until such time as states enact appropriate safeguards, and congratulate Special Rapporteur Achiume on joining that call.
2. Gender mainstreaming and representation along racial, national and other intersecting identities requires radical improvement at all levels of the tech sector.
   The structural racism and discrimination that the report identifies as endemic to the field of technology (just as it is in many if not all facets of societies around the globe) cannot be remedied if the teams conceiving of, building, and promoting technological solutions do not understand and represent the concerns of those who will be impacted by them. When implemented, these demographic changes must be meaningful: past “diversity” and “inclusivity” efforts in the industry have often been mere tokenizations of underrepresented groups. Meaningful improvement will mean significant changes to industry power structures, funding flows and models, cultural changes in the workplace, and reevaluation of existing and future product lines that may be employed to target racially marginalized communities and other vulnerable populations.[1]
3. Technologists cannot solve political, social, and economic problems without the input of domain experts and those personally impacted.
   The past several decades have been dominated by “techno-chauvinism,” the idea that technology alone can solve social problems.[2] But as Special Rapporteur Achiume rightly notes, no algorithmic model, no matter how “perfect,” will solve for centuries of inequality. For this reason, tech design and development must include domain experts (including those with first-hand experience alongside those with professional or academic expertise) in a consequential way. Such experts cannot be briefly consulted late in a product’s development, or worse, after its negative impacts are already being felt, but integrated into the design process. Furthermore, they must be compensated for their contributions. This is especially important in sensitive sectors such as education, criminal justice, and social services, where past technologies have been adopted in spite of flaws and biases that were facially apparent to experts in the respective field. Moreover, such experts can help to identify how well-meaning technologies may be distorted or misused in practice, exacerbating social inequities and harms rather than actually addressing community needs.
4. Access to technology is as urgent an issue of racial discrimination as inequity in the design of technologies themselves.
   Digital divides illustrate the intersectionality of discrimination in technology, and the COVID-19 pandemic has thrown them into sharp relief. The global South lags the global North in the digital infrastructure that can be employed to provide remote access to medical care, education, and more, but even within so-called “developed” countries, poor and otherwise marginalized communities lack access to these necessary tools. Digital divides are also present within global South states: we have not reached Sustainable Development Goal target 9(c), to bring all those in least developed countries online by 2020. As societies’ reliance on technological interventions increases over time, these divides will deepen in significance and pose life-or-death threats to people around the world.
5. Representative and disaggregated data is a necessary, if not sufficient, condition for racial equity in emerging digital technologies, but it must be collected and managed equitably as well.
   We welcome Special Rapporteur Achiume’s call for “States to collect, compile, analyse, disseminate and publish reliable statistical data disaggregated on racial or ethnic grounds.” However, despite important advances in data protection laws globally, adequate standards for non-extractive data collection and governance have not yet been disseminated. The history—and present—of data collection is rife with examples of the exploitation of marginalized populations, and the data that is thus taken is then generally managed in such a way that it can too often be used to further target and victimize. We would welcome, and gladly participate, in an effort to develop such standards, which should include both measures to ensure that data is truly representative and respectful, not reinforcing existing racial and other hierarchies; address the power dynamics between data collectors and those whose data is collected, including around meaningful consent; and provide safeguards to prevent such data from being used by malign actors to further oppress marginalized people.
6. States as well as corporations must provide remedies for racial discrimination, including reparations.
   The signatories followed with interest Special Rapporteur Achiume’s previous report on the centrality of reparations in remedying the many years of socioeconomic oppression that endured during and continue after colonialism and slavery. In this report, the Special Rapporteur rightly notes that states and companies who have been involved with or overseen the development and deployment of emerging digital technologies that introduced novel instances of racial discrimination or deepened existing inequalities are obligated to participate in remedial processes and provide adequate compensation. Given the interplay between economic privilege and access to technology, as well as the rooms in which it is designed and regulated, we wish to underscore the link between these reports, and to encourage the tech giants whose record profits result at least in part from this inequality to consider acting on their public commitments to non-discrimination. This includes meaningful engagement by states and tech companies to evaluate their role in maintaining racial power structures and taking necessary steps to dismantle them.

[1] https://onezero.medium.com/the-seductive-diversion-of-solving-bias-in-artificial-intelligence-890df5e5ef53

[2] https://www.theverge.com/2018/5/23/17384324/meredith-broussard-artifical-unintelligence-technology-criticism-technochauvinism

Karine Gentelet, Université du Québec en Outaouais
Professor Ellen P. Goodman, Rutgers Law School
Dr. Sarah Myers West, AI Now Institute/ Data Justice Advocate / HCI Researcher
Dr. Stephen C. Rea, Colorado School of Mines
Dr. Joy Lisi Rankin, AI Now Institute, New York University
Stefaan G. Verhulst, Co-Founder, The GovLab, NYU
Charlie Martial NGOUNOU
Usama Khilji, Bolo Bhi
Ahmed Ansari, NYU
Avis MOMENI/ Civil Society actor
damian loreti, Individual member of APC
Jane Anderson, ENRICH and New York University
Molly Land, Human Rights Institute, University of Connecticut
Mariana Valente
Meredith Broussard, New York University; author of “Artificial Unintelligence: How Computers Misunderstand the World”
Jessica Fjeld, Harvard Law School Cyberlaw Clinic at the Berkman Klein Center for Internet & Society
Mara Mills, Associate Professor, New York University
Ephraim Percy Kenyanito, Global Voices
Parviz Khazaei
Andrew Sporle, The University of Auckland, iNZight Analytics Ltd.
Dr Monique Mann, Senior Lecturer in Criminology, Deakin University
Gayathry Venkiteswaran, media freedom activist, Malaysia
Sabine Poeschmann
Kate Sim, Oxford Internet Institute
John Lunsford
Dr Emily Petherick Loughborough University
Aung Zaw Min
MUHEEB SAEED, MEDIA FOUNDATION FOR WEST AFRICA
Daniel Kreiss, UNC Center for Information, Technology, and Public Life
Zeerak Waseem, University of Sheffield
Tonia Sutherland, PhD, Director SOURCE Hawaiʻi, University of Hawaiʻi at Mānoa, Advidory Board, Center for Critical Race + Digital Studies
Janna Huang, UC Berkeley Sociology
Lucy Bernholz, Digital Civil Society Lab, Stanford University
Resident Fellow, Information Society Project at Yale Law School
Vibranium Health Equity
Leyla Gayibova, Teaching Assistant at The Hague University of Applied Sciences
Ben Green, Harvard University
Samuelson-Glushko Professor of Law, University of Ottawa
University of Livingstonia, Malawi
Juliet Nanfuka, CIPESA
Gürkan Özturan, Journalist, Digital Rights & Liberties Analyst
Dr. Farida Vis, Media Department, Manchester Metropolitan University (UK), Director Visual Social Media Lab, Advisory Board Member, Center for Critical Race & Digital Studies
Dr. Simon Faulkner, Department of Art and Performance, Manchester Metropolitan University, United Kingdom
Gaëtan Goldberg (Data Rights)
Mariana Fossatti, APC
Dr. Baobao Zhang, Cornell Society of Fellows; Berkman Klein Center for Internet & Society, Harvard University
Christelle A.
Abebe Chekol, ICT4D Expert
Cynthia Khoo, Research Fellow, Citizen Lab
Rediet Abebe, Harvard Society of Fellows
Daniel Shiffman, New York University, Tisch School of the Arts, ITP/IMA
Lynda Osborne, M.S., Ph.D. Candidate, Educational Psychologist, Human Factors Engineer, Founder of the Consumer Tech Innovation Festival (CTIF) and the conference on AI Prospects, Perils & Commerce