V2-2272

Defining a framework for ensuring public confidence in AI systems and their use

Artificial Intelligence (AI) technology has achieved a number of breakthrough successes, particularly in the last 10 years, especially with the highly successful application-relevant machine learning methods. These methods are now successfully used in many application areas such as: image recognition, machine translation, robotics, automatic text and speech generation, medical diagnosis and decision making in healthcare, automated decision making, recommender systems, automatic discovery in science, new drug discovery, etc. However, some aspects of AI technology, such as deep neural networks, have aroused public suspicion. Some of the reasons for distrust in the use of AI include: the incomprehensibility of decisions made by AI; the appearance of bias or discrimination in decisions proposed by AI (e.g. in the judiciary, employment, healthcare); the tendency to manipulate people through social media; threats to democracy – the possibility of election manipulation; recommendation systems in social media that lead to the potentiation of harmful and hateful content across social media; the use of AI in killer autonomous weapons.

The above reservations about the use of AI have led to a number of initiatives focusing on the ethical aspects of AI and concerns about some uses of AI. These initiatives have contributed to the definition of a framework for trustworthy AI and principles that would pave the way for trust in the use of AI through regulations and laws. However, the general public’s attitude towards the use of AI often remains highly critical.

One of the main criticisms in this respect is described in the public domain by the phrase algorithmic bias. The following two examples are probably the most widely criticised in the public domain as examples of questionable applications of AI: (1) the COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) system for assisting in judicial decision-making and (2) the Amazon job applicant assessment system. In both controversial cases, the main complaint is that the AI system suggests biased decisions. The alleged bias is related to discrimination on the basis of race or gender. These two cases and similar cases are publicly regarded as unacceptable uses of AI, often associated with phrases such as “algorithm bias”, “AI bias” and “machine learning bias”.

Another shortcoming of some AI systems, which is very often mentioned by the general public and also in political circles, is the incomprehensibility of the decisions of the AI system, or the lack of a humanly comprehensible explanation for these decisions.

In addition to these two problems, which lead to serious reservations about the acceptability of using AI, there are a number of other reservations, which are listed above. Some of these are also being professionally and intensively studied in the field of AI itself, and the negative public perception does not always seem to be fully justified. In this targeted research project, we will assess what the actual risks associated with the general concerns are and what the public perception of them is, both globally and in Slovenia. We will propose a framework for building trustworthy AI, including guidelines for topical amendments to legislation where we think they are needed. We will also assess the effectiveness of technical innovations in AI methods aimed at consolidating trustworthy AI and addressing concerns. In doing so, we will also contribute some of our own innovations in terms of explainable AI and educating the general public about AI in everyday life.

We will address the following concerns and questions regarding the use of AI:

• The incomprehensibility of the system’s decisions – how and why did the programme make the decisions it did? This problem is also known as the black box problem or the explainability problem.
• The bias of machine learning decisions. Actually, it is mainly about the impression of bias that users have. But it is hardly really algorithm bias, as it is often called in general circles by commentators; it is at most the bias of the data from which the programme has learned. The impression of bias can also arise from mathematically and statistically well-founded decisions in machine learning. For example, a simple Laplace probability estimate can explain why minority groups are sometimes treated differently from majority groups.
• Deep fake – AI as a tool for fabricating untruths that spread on social media, e.g. fake videos with deep neural networks.
• Collection of personal data – What happens to this data, where are the limits of what is acceptable?
• The question of liability for AI system failures – who is formally responsible for the wrong decisions that can occur when using complex AI systems?
• Manipulation of people across social media with recommendation systems to recommend content tailored to the profile of the targeted user. Whether for advertising, to change opinions across the network, or also for political advertising and influence.
• Spreading harmful and dangerous content via AI (e.g. violent or intolerant) across social media.
• Vpliv na demokratične procese – vpliv na volilce z uporabo UI in zlorabi osebnih podatkov, tudi takih, ki so pridobljeni nelegalno ali pa z nezavednim soglasjem uporabnika. Oširno je ta problem obravnavan v posebni številki revije Scientific American z naslovom Can democracy survive big data and AI?
• Influencing democratic processes – influencing voters through the use of AI and the misuse of personal data, including that obtained illegally or with the unconscious consent of the user. This issue is discussed at length in a special issue of Scientific American entitled Can democracy survive big data and AI?
• Big language models such as GPT-3, GPT-4 and ChatGPT. There are at least two problems here: (1) the indistinguishability of who wrote the text, human or computer; the potential for misuse is enormous, and this can lead, among other things, to unstoppable information pollution; (2) the use of big language models for automatic programming. Here there is no guarantee as to the correctness of the software so generated, catastrophic errors are possible when such software is actually used; there are no formal specifications for such software, not even entirely satisfactory informal ones, what does the software actually do?
• The question of authorship of patents and AI-generated artwork. For example, who is the author of the paintings automatically generated by the deep anechoic net after learning from examples of paintings by a famous painter?

Project goal

C1: Identify and analyse the impact of different characteristics of an AI system such as technological characteristics (robustness, explainability, transparency, models used), development methodologies used, existence of and compliance with relevant standards, type of licensing (e.g. open source, proprietary software) that are relevant for individual trust in AI;

C2: Identification and analysis of the impact of different ethical principles and fundamental rights on individual trust in AI; analysis of how specific concerns and issues not identified above relate to the public’s perception of the use of AI in the light of these ethical principles.

C3: Identify and analyse the impact of cultural and societal characteristics such as economic stability, social vulnerability, access to education, adequate legislation to ensure legal certainty and predictability on individuals’ trust in AI;

C4: Identification and analysis of individual status characteristics such as education, employment, age, ICT knowledge and skills on individuals’ confidence in AI;

C5: Establish a framework for analysing and monitoring the state of public trust in AI based on the interdependence and influence of the factors identified above, i.e. the characteristics of AI solutions, ethical principles, social and cultural characteristics, and the characteristics of the individual’s status;

C6: Creation of a demonstration version of the Orange Machine Learning System specifically designed for general education on the opportunities and risks of machine learning.

Project funding:

The project has received funding by the Slovenian Research and Innovation Agency.