Australian Human Rights Commission Human Rights in the Digital Age: Additional Material Submitted to the UN Global Digital Compact, 30 April 2023 52. AI systems may unintentionally produce discrimination in the employee vetting process. For instance, Amazon used an AI software that was designed to review resumes and determine which applicants Amazon should hire.32 The algorithm systemically discriminated against women applying for technical jobs, such as software engineer positions. This is because the existing pool of Amazon software engineers were by majority male, and as such, the new software was fed data about those engineers’ resumes.33 The practice of directing software to discover resumes that harbor similarities to resumes in a training data set will inevitably reproduce the demographics of the existing workforce.34 53. Another example of algorithmic bias was when, in 2019, a study discovered that a clinical algorithm used by many hospitals in the US to determine which patients required extra medical care produced racial bias.35 The algorithm was trained on past data on healthcare spending, which reflects a trend whereby black patients have less income to spend on their healthcare as compared with white patients - a result of systemic wealth and income disparities.36 As such, the algorithm’s outputs reflected a discriminatory result whereby white patients required more medical care than black patients.37 54. Such examples highlight why AI requires greater regulation, in the interests of increasing transparency and preventing unfairness and unlawful discrimination in algorithmic decision-making. This is especially the case given the difficulty of applying anti-discrimination laws to complex AI systems.38 The Commission emphasises its 2020 technical paper Using artificial intelligence to make decisions: Addressing the problem of algorithmic bias which considers algorithmic bias in greater detail. 4.2 AI Safety Commissioner 55. In responding to how AI can be regulated countries should, at first instance, modify their existing laws, regulations and regulatory bodies in a manner which better allows them to respond to the risks posed by AI. However, this is a short-term response, and there will ultimately need to be an AI-specific statutory body to respond to new and emerging risks in this area. 56. The Commission notes recommendations 22 and 23 of its Final Report.39 Recommendation 22 stated: The Australian Government should establish an AI Safety Commissioner as an independent statutory office, focused on promoting safety and protecting human rights in the development and use of AI in Australia. The AI Safety Commissioner should: 13