violated his due process rights because the COMPAS reports provide data relevant only to particular groups and because the methodology used to make the reports is secret. Therefore, the use of the COMPAS assessment infringed on both his right to an individualised sentence and his right to be sentenced on accurate information. The defendant also complained that it was unconstitutional because it took gender into account. The Supreme Court ruled that a trial court’s use of an algorithmic risk assessment in sentencing did not violate the defendant’s due process, highlighting that the COMPAS report was not the sole basis of the sentencing decision. However, the Judge added that judges must proceed with caution and expressed scepticism when using such risk assessments. To ensure that judges weigh risk assessment appropriately, the court prescribed both how these assessments must be presented to trial courts and the extent to which judges may use them. The court explained that risk scores may not be used “to determine whether an offender is incarcerated” or “to determine the severity of the sentence.”237 Therefore, judges using risk assessments must explain the factors other than the assessment that support the sentence imposed. Five written warnings must also be provided to judges when COMPAS assessments are used.238 Concerns regarding the inherent risk of bias that arises from algorithmic risk assessments in the criminal justice sector were raised by U.S. Attorney General, Eric Holder in 2014.239 Furthermore, in an investigation by ProPublica in 2016 found that COMPAS assessments for more than 7,000 arrestees in Florida was based on an algorithm that was biased against African 237 State v. Loomis 881 N.W.2d 749 (Wis. 2016) at 769. 238 Ibid. 239 In 2014, then U.S. Attorney General Eric Holder raised concern about algorithms that produce risk assessments that seek to assign the probability of individual’s likelihood of committing future crimes: “Although these measures were crafted with the best of intentions, I am concerned that they inadvertently undermine our efforts to ensure individualized and equal justice… they may exacerbate unwarranted and unjust disparities that are already far too common in our criminal justice system and in our society. See https://www.justice.gov/opa/ speech/attorney-general-eric-holder-speaks-national-association-criminal-defense-lawyers-57th. Americans.240 AI systems challenge the right to privacy because they depend on ingesting as much data as possible, a methodology which is adverse to privacy.241 AI’s capacity for prediction and inference adds to these concerns. This has led to calls for professional and legal ethical codes to be developed that govern the design and application of AI technologies and apply to the activities of both governments and private sector organisations and entities.242 The European Union has introduced the first piece of legislation to address algorithmic discrimination in the European General Data Protection Regulation. This recognises the effect of algorithmic decision making on fundamental rights and addresses algorithmic discrimination, which occurs when an individual or group receives unfair treatment as a result of algorithmic decision-making. The ERDP addresses three principles: • Data sanitisation – removal of specific categories from data sets; prohibition against “processing of data revealing racial or ethnic origin” and other “special categories” (Article 9); and prohibits decisions based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significant effects against him or her” that is “based on the special categories of information referred to in Article 9” (Article 22) • Data transparency – right to an explanation where entitled to information about logic involved and envisaged consequences (Article 13, 14) 240 The report by Pro Publica, a non-profit newsroom based in the US, investigated whether the use of algorithms by courts and law enforcement to predict the likelihood of a defendant re-offending were bias against African Americans. It obtained the risk scores assigned to more than 7,000 people arrested in Florida in 2013 and 2014 and checked to see how may were charged with new crimes over the next two years. They found that only 20 percent of the people predicted to commit violent crimes actually went on to do so. Significant racial disparities were found to exist, with the formula more likely to falsely flag black defendants as future criminals, wrongly labelling them this way at almost twice the rate as white defendants. See https://www.propublica. org/article/machine-bias-risk-assessments-in-criminal-sentencing 241 Ibid p 28. 242 Ibid pp 32-34. 45