Artificial intelligence and racial bias in data A comprehensive antiracism framework must take into account the emerging technologies which bring new challenges of racial bias into our everyday lives. […] Biases within such technologies poses the threat of misidentification, discrimination, and wrongful prosecution. — submission from the Asian Australian Alliance, NARF project, May 2021 – April 2022 The emergence of artificial intelligence (AI) has seen a rise in concerns of algorithmic bias in AIled decision-making processes in both the public and private sectors. Although AI has aims to progress technological capabilities, previous Commission research, as well as consultation and submission participants on the National Anti-Racism Framework project, raised serious concerns about the possibility of algorithms replicating the racial bias in data and further entrenching discriminatory practices. AI-informed risk assessment is a form of decision-making that is increasingly common within the criminal justice system, immigration departments, and other government services like healthcare. When flawed data sets, such as historical data affected by racial bias or prejudice, are used to build and train an assessment tool or create projections, the algorithms will produce biased outcomes that reinforce human prejudice and systemic racism in the real world, to the detriment of negatively racialised groups.48 An example of structural racism being reinforced through AI algorithms is the Suspect Target Management Plan used by the NSW Police to identify potential criminal offenders. The tool disproportionately labels First Nations people, especially young First Nations people, as potential criminal offenders. This is due to the tool’s ability to target a race related factor based on historical discriminatory statistics, namely criminal offence data in which First Nations people are disproportionately overrepresented, as part of its assessment.49 Discriminatory decision-making arising from AI-informed risk assessments is difficult for individuals to navigate through anti-discrimination laws. The Commission’s previous research into human rights protections in the area of emerging technologies also indicates that current laws leave the variables within, and reasoning behind, these machine decisions unchecked, and fall short in addressing the potential negative effects of AI technology. 50 In view of inadequate protections, consultation participants called for greater accountability and transparency around the use of big data, including in relation to the connections between data collected, and its use by different governments and private and public services. Participants also shared the need for capacity building and defined strategies countering racial bias in emerging technologies. Participants’ initial feedback about AI supports the Commission’s proposals in the Human Rights and Technology Report, which recommends that Australian governments incorporate international human rights frameworks into services driven by AI and create accountability mechanisms to redress harm caused by AI technology. This includes allowing challenges to an AIinformed decision, providing opportunities to obtain explanations for decisions made, and ensuring relevant testing is conducted to prevent algorithmic bias. 51 National Anti-Racism Framework Scoping Report 2022 | 97