Explainable Deep Learning AI: Methods and Challenges

Preface

Georges Quénot; Jenny Benois-Pineau; Dragutin Petkovic; Romain Bourqui     

If you can't explain it simply, you don't understand it well enough.

Albert Einstein, Physicist

Artificial Intelligence (AI) techniques, especially those based on Deep Learning (DL), have become extremely effective on a very large variety of tasks, sometimes performing even better than human experts. However, they also have a number of problems: they generally operate in mostly opaque and/or intractable ways, their very good performance is only statistical and they can fail even on apparently obvious cases, they can make biased decisions, and they can be quite easily manipulated through adversarial attacks, to cite a few. These limitations prevent their adoption in applications of great economic or societal interest, especially for critical or sensible applications like autonomous driving, medical diagnosis, or loan approvals.

Considering this, a lot of research has been conducted in order to increase the trustworthiness of DL-based AI systems by providing explanations understandable by human users for the decisions made by these systems. The aim of this book is to present recent and original contributions covering the main approaches in the domain of explainable DL, either for expert or for layman users. Two main types of approaches are presented: the post hoc or model agnostic ones, in which the operation of an already available black box system is modeled and explained, and the intrinsic ones, in which systems are specifically designed as white boxes with an interpretable mode of operation.

This book is a follow-up of a very successful Workshop Explainable Deep Learning-AI which we have organized at ICPR'2020 IAPR and IEEE conference making a large tour of ongoing research in the field.

The question of the explainability is addressed in a variety of media domains, including images, text, temporal series, and tabular data. The difficult and important question of the evaluation of explanation methods is also specifically addressed in two chapters. Finally, still open challenges are mentioned in the concluding chapter.

LIG/CNRS, LABRI/University of Bordeaux (France), San Francisco State University (USA), LABRI/University of Bordeaux (France)

April 5, 2022

Chapter 1: Introduction

Jenny Benois-Pineaua; Dragutin Petkovicb    aUniv. Bordeaux, CNRS, Bordeaux INP, LaBRI, UMR 5800, Talence, France

bSan Francisco State University, San Francisco, CA, United States

Abstract

The recent focus of Artificial Intelligence (AI) on the supervised learning approaches, and particularly on Deep Learning (DL) / AI, resulted in considerable increase of performance of AI systems, making them useful in real-world applications. It has also raised the question of the trustworthiness and explainability of their predictions for decision making. In critical application domains, such as healthcare and security, the trustworthiness of AI systems is the sine qua non condition for their adoption, but, as it is well known, decisions made by DL systems are very hard to explain or make transparent. To make these AI DL systems transparent and understandable for the user, a new research field has emerged, namely Explainable Artificial Intelligence (XAI). In this chapter we will introduce the rationale of this book and make a short overview of its chapters, which present new methods for explainable DL and also show their applications in various domains where AI-based decision making is desirable.

Keywords

Explainable AI; Tristworthy AI; Deep Learning

Chapter points

•  In this chapter we first give the rationale of the book.

•  The content of each chapter will be shortly introduced.

We are witnessing the emergence of an AI economy and society where AI technologies are increasingly impacting many aspects of business as well as everyday life. We read with great interest about the recent advances in AI medical diagnostic systems, self-driving cars, the ability of AI technology to automate many aspects of business decisions like loan approvals, hiring, policing, etc. However, as evident by recent experiences, AI systems may produce errors, can exhibit overt or subtle bias, may be sensitive to noise in the data, and often lack technical and judicial transparency and explainability. These shortcomings have been documented in the scientific literature, but also, importantly, in the general press (accidents with self-driving cars, biases in AI-based policing, hiring, and loan systems, biases in face recognition systems for people of color, seemingly correct medical diagnoses later found to be made due to wrong reasons, etc.). These shortcomings are raising many ethical and policy concerns not only in technical and academic communities, but also among policymakers and general public, and will inevitably impede wider adoption of this potentially very beneficial technology. These broad concerns about AI are often grouped under an umbrella area of AI Ethics and Values or Trustworthy AI. The technical community, potential adopters, popular media, as well as political and legal stakeholders, have recognized the problem and have begun to seek and ask for solutions. Many very high level political and technical bodies (e.g., G20, EU expert groups, Association of Computing Machinery in the USA), as well as major AI companies, have identified Trustworthy AI as a critical issue. The technical and academic community are also addressing these issues by dedicated conferences and workshops and academic centers. The governing bodies are issuing guidance and even laws like EU GDPR with right to know clauses for AI-driven decisions.

The problems related to Trustworthy AI are complex and broad, and encompass not only technical issues but also legal, political, and ethical ones. Trustworthy AI concerns seem to have some common components, such as bias/discrimination, need for human understanding and control, ethics, as well as judicial and human-centric transparency and explainability. It is also a common belief that better methods for auditing and even certification of AI systems for Trustworthiness are going to be critical for its broader adoption in business and society.

One way to tackle broad problems like Trustworthy AI is to first work on its key components. We hence devote this book to AI explainability, which is what we believe of the critical issue in achieving Trustworthy AI – after all, how can one audit/certify, trust, and legally defend (that will come, too!) something one does not understand? More specifically, this book addresses explainability of Deep Learning (DL) AI technologies, which are very popular and powerful, but at the same time notoriously hard to explain. At a high level, explainability of AI systems should allow human users to gain insights into how they make their decisions. The human users of Explainable AI have to include not only AI experts but also domain experts (often the key adopters), as well as general public. AI explainability can be model based, where it offers insights on how the AI system works on a collection of data as a whole (e.g., the whole training database), and sample based, where it offers insights into how the trained Machine Learning system classifies a specific data sample (why was I rejected for the loan?). The latter has been shown to be critical in determining user trust in AI systems. AI explainers can be agnostic to AI methods they try to explain, or direct (tied to a specific AI algorithm). AI systems that are explainable achieve many benefits, e.g., they could increase user trust and adoption, improve AI quality control and maintenance, make them legally more transparent, and possibly even offer new insights into the analyzed domain. The need for explainability does not exclude the usefulness of black-box models since they are often tried first and may serve, among other things, to point to ultimate achievable accuracy. Explainable systems can also be used for development, verification, and testing of AI systems including otherwise hard to explain black box systems.

Challenges in achieving explainability of DL AI systems are very significant. Due to their structure (multiple layers of a very large number of interconnected weighted connections), they are extremely hard to explain compared to, for example, powerful ensemble tree based systems like Random Forests. The challenges for explainable DL include algorithms to extract explainable information from trained DL systems, ways to present and visualize explanation information such that it is understandable to target human users (experts or not) at the model as well as sample level, and methods for evaluation of such systems which have to involve target human users, thus making them time consuming and expensive.

This book covers the latest and novel contributions to all aspects of explainable DL, from area overview to specific approaches, including comprehensive evaluations of usefulness for human users. It comprises 13 contributing chapters which present methodological aspects, together with application examples.

Chapter 2 introduces the reader to the concepts, methods, and recent developments in the field of explainable AI. After discussing what it means to explain in the context of machine learning and presenting useful desiderata for explanations, a brief overview of established explanation techniques is provided, mainly focusing on the so-called attribution methods which explain the model's decisions by post hoc assigning importance scores to every input dimension (e.g., pixels in images). Furthermore, recent developments in XAI are discussed. In addition, the author presents ways to use explanations to effectively prune, debug, and improve a given model, as well as the concept of neuralization which easily allows one to transfer XAI methods specifically developed for neural network classifiers to other types of models and tasks. The chapter concludes with a discussion of the limits of current explanation methods and promising future research directions.

In Chapter 3 the authors make use of information visualization approaches to present a method to help in the interpretability of trained deep neural networks. By depicting both their architecture and their way to process the different classes of a testing input dataset, the moment (i.e., the layer) where classes are being discriminated becomes visible. Similarly, they show how it is possible to detect degradation in the classification process, mainly because of the usage of an oversized network for a simpler classification task. The authors conduct their experiments on well-known Deep NN architectures, such as LeNet5 and VGG16 networks, when using popular MNIST and Fashion-MNIST data sets. Results show a progressive classification process which is extended to an improvement on an oversized network to solve a simpler task.

In Chapter 4 the authors address the family of explanation methods by perturbation. These methods are based on evaluating the effect that perturbations applied to the input induce on the model's output. We note that the authors are in particular interested in image classification. In this chapter, they discuss the limitations of the existing perturbation-based approaches and contribute with a perturbation-based attribution method guided by semantic segmentation of images. This method inhibits specific image areas according to their semantic meaning, associated to an assigned semantic label. Hereby, perturbations are linked up with this semantic meaning and a complete attribution map is obtained for all image pixels. The potential capabilities of this attribution scheme are exemplified by automatically arranging image areas into semantic sets according to their relevance, irrelevance, and distracting potential for scene recognition of the semantic label assigned to them.

Chapter 5 presents a further development of recently introduced Feature Understanding Method (FEM) which explains DNN decisions also in the context of image classification tasks. The authors call the method Modified FEM. It belongs to the family of the so-called white-box methods. It is based on the analysis of the features in the last convolution layer of the network with further backpropagation to identify the image pixels which contributed to the decision the most. The method explains the decision of a trained network for a specific sample image. The application example of the method is explanation of the network trained for the classification of chest X-ray images for the recognition of COVID-19 disease. This chapter covers the intensive research in the XAI field despite its relatively rich set of methods which have been previously developed.

Chapter 6 is an example of the use of XAI methods in a clinical field. The authors do not propose new explainers. Instead, they use methods previously proposed in the literature, such as Backpropagation, Guided Backpropagation, and Layerwise Relevance Propagation. They hypothesize that the agreement across these methods is an indication of the robustness of the results in the problem of stratification of multiple sclerosis on brain images. The voxels of the input data mostly involved in the classification decision are identified and their association with clinical scores is assessed, potentially bringing to light brain regions which might reveal disease signatures. Indeed, presented results highlight regions such as the Parahippocampal Gyrus, among others, showing both high stability across the three visualization methods and a significant correlation with the Expanded Disability Status Scale (EDSS) score, witnessing in favor of the neuro-physiological plausibility of these findings.

In Chapter 7 as in Chapter 4, the authors focus on perturbation based methods which use some modifications of the initial signal (more often, in some neighborhood around it) and measure how the classification result changes. They propose a method recursively hides image parts to get an explanation for a particular decision of the black-box classifier. The core of this method is the division of the image being classified into separate rectangular parts, followed by the analysis of their influence on the classification result. Such divisions are repeated recursively until the explanation of the classification result is found, or the size of parts is too small. As a result, the pair of images with complementary hidden parts is discovered: the first image preserves both the most valuable parts and the classification result of the initial image. The second shows the result of hiding the most valuable parts of an initial image that leads to the different classification. Such a representation allows humans to see which parts of the image are significant for the particular decision and confirms that classification is not successful without these parts.

Chapter 8 focuses on the importance of trained convolutional filters in the convolutional neural networks. Removing some of them, the authors track classification results and thus also explain the network. Filter removal can lead to the increased accuracy, which explains the chapter title Remove to improve.

Chapter 9 is interesting in the sense that it combines rule-based methods which are naturally explainable prediction models, with CNNs. The rule mining techniques such as SBRL, Rule Regularization, and Gini Regularization are adapted to produce an interpretable surrogate model, which imitates the behavior of the CNN classifier on time series data. This makes the decision process more comprehensible for humans. In the experiments the authors evaluate three methods on the classification task trained on AIS real-world dataset and compare the performance of each method in terms of metrics such as F1-score, fidelity, and the number of rules in the surrogate list. Besides, they illustrate the impact of support, and bins (number of discrete intervals), parameters on the model's performance along with the strengths and weaknesses of each method.

Chapter 10 addresses an extremely important and still open research question of the evaluation of explanation methods. It presents two protocols to compare different XAI methods. The first protocol is designed to be applied when no end users are available. An objective, quantitative metric is being compared to objective data expert's analysis. The second experiment is designed to take into account end-users feedback. It uses the quantitative metric applied in the first experiment, and compares it to users preferences. Then, the quantitative metric can be used to evaluate explanations, allowing multiple explanation methods to be tried and adjusted, without the cost of having systematic end-users' evaluation. The protocol can be applied on post hoc explaining approaches, as well as to self-explaining neural networks. The application example used is natural language processing tasks.

Chapter 11 is application oriented in the field of cybersecurity. As machine learning approaches are used for malware detection, XAI approaches allow for detecting the most characteristic features both in the static and dynamic analysis frameworks, which characterize malware. In this perspective, the authors focus on such challenges and the potential uses of explainability techniques in the context of Android ransomware, which represents a serious threat for mobile platforms. They present an approach that enables the identification of the most influential features and the analysis of ransomware. Their results suggest that the proposed methods can help cyber threat intelligence teams in the early detection of new ransomware families and could be extended to other types of malware.

The authors of Chapter 12 are interested in explainability in medical image captioning tasks. Image captioning is the task of describing the content of the image using textual representation. It has been used in many applications such as semantic tagging, image retrieval, early childhood learning, human-like robot–robot interactions, visual question answering tasks, and medical diagnosis. In the medical field, automatic captioning assists medical professionals in diagnosis, disease treatment, and follow-up recommendations. As a general trend, deep learning techniques have been developed for image captioning task. They rely on encoder–decoder models which are black boxes of two components cooperating to generate new captions for images. Image captioning builds a bridge between natural language processing and image processing, making it difficult to understand the correspondence between visual and semantic features. The authors present an explainable approach that provides a sound interpretation of the attention-based encoder–decoder model for image captioning. It provides a visual link between the region of medical image and the corresponding wording in the generated sentence. The authors evaluate the performance of the model and provide samples from the ImageCLEF medical captioning dataset.

Chapter 13 is the second one which explicitly tackles the question of the evaluation of explainers. This chapter elaborates on some key ideas and studies designed to provide post hoc explanations-by-example to the problem of explaining the predictions of black-box deep learning systems. With a focus on image and time series data, the authors review several recent explanation strategies – using factual, counterfactual, and semifactual explanations – for Deep Learners. Several novel evaluations of these methods are reported, showing how well these methods work, along with representative outputs that are produced. The chapter also profiles the user studies being carried out on these methods, discussing the pitfalls that arise and results found.

Up to now a wide variety of XAI methods have been proposed. It is not always obvious for an AI practitioner which of them is the most applicable to the problem at hand, as the theoretical analysis of these methods and the conditions of their applicability remain rare. One of the popular methods in the family of black-box XAI approaches is Locally Interpretable Model-agnostic Explanation (LIME). It consists in masking presegmented regions in the data (images) and tracking the score changes, thus determining the regions important for classification. The exact operation of LIME, though, is often overlooked by practitioners, sometimes at a great expense. One example in particular is the nature of the sampling process, which can lead to misleading explanations if not adapted to the data at hand. Another point is the role of LIME's hyperparameters, of which the provided explanations depend in a complicated manner. In Chapter 14, the author summarizes some recent theoretical work focused on LIME with the goal to propose useful insights into the method.

Finally, Chapter 15 concludes the book. We review the content of the book providing insights about the main concepts presented in the previous chapters. We also give two important directions for future work: the evaluation of XAI methods and the enrichment of the explanation methods with semantics in order to foster trust to nonexperts.

Chapter 2: Explainable deep learning: concepts, methods, and new developments

Wojciech Sameka,b,c    aTechnical University of Berlin, Department of Electrical Engineering and Computer Science, Berlin, Germany

bFraunhofer Heinrich Hertz Institute, Department of Artificial Intelligence, Berlin, Germany

cBIFOLD - Berlin Institute for the Foundations of Learning and Data, Berlin, Germany

Abstract

Explainable AI (XAI) is an emerging research field bringing transparency to highly complex and opaque machine learning (ML) models. In recent years, various techniques have been proposed to explain and understand ML models, which have been previously widely considered black boxes (e.g., deep neural networks), and verify their predictions. Surprisingly, the prediction strategies of these models at times turned out to be somehow flawed and not aligned with human intuition, e.g., due to biases or spurious correlations in the training data. Recent endeavors of the XAI community aim to move beyond the mere identification of these flawed behaviors, toward the integration of explanations into the training process in order to improve model efficiency, robustness, and generalization. This chapter introduces the reader to concepts, methods, and recent developments in the field of explainable AI. After discussing what it means to explain in the context of machine learning and presenting useful desiderata for explanations, we provide a brief overview over established explanation techniques, mainly focusing on the so-called attribution methods which explain the model's decisions by post hoc assigning importance scores to every input dimension (e.g., pixel). Furthermore, we discuss recent developments in XAI, in particular we present ways to use explanations to effectively prune, debug, and improve a given model, as well as the concept of neuralization which easily allows one to transfer XAI methods specifically developed for neural network classifiers to other types of models and tasks. The chapter concludes with a discussion of the limits of current explanation methods and promising future research directions.

Keywords

Explainable AI; Black box models; Desiderata for explanations; Attribution methods; XAI-based model improvement; Neuralization

•  Why do we need explanations? From black-box models to explainable AI.

•  What to explain? Definitions and desiderata of explanations.

•  How to explain? Overview over post hoc explanation methods.

•  Beyond explaining. From explanations to better models.

Acknowledgement

This work was supported by the German Federal Ministry of Education and Research (BMBF) under grants 01IS18025A, 01IS18037I, 031L0207C, and 01IS21069B. Furthermore, this work has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement no. 965221.

2.1 Introduction

In the last decade, deep learning has revolutionalized the research in the field of artificial intelligence (AI) and is now slowly transforming the world we live in. Smart services, intelligent devices, and advanced analysis tools are becoming more and more available and are increasingly being used. Google Translate and Amazon's Alexa are popular services demonstrating the power of today's deep learning-centered AI. While the AI revolution has started in the field of computer vision and natural language processing, it is now also transforming many other application domains, including medicine, signal processing, and communications. This trend will intensify as more and more methods augmented with learning capabilities outperform traditional algorithms relying on hand-crafted features. Examples can be found in image compression (Toderici et al., 2017), physical layer communications (Qin et al., 2019) or ECG-based arrhythmia classification (Hannun et al., 2019).

Due to the complexity of commonly used deep neural networks, the question of interpretability is of central importance. The recent rise in the field of Explainable AI (XAI) (Samek et al., 2019) shows the growing need to understand the reasoning of these powerful learning machines. The reasons for making machine learning models explainable are diverse and range from more trust and verification capabilities (Holzinger et al., 2019), over the generation of deeper (scientific) insights (Roscher et al., 2020) and legal requirements (Hacker et al., 2020), to the ability to debug the model and identify Clever Hans predictors¹ (Lapuschkin et al., 2019). One spectacular example of the last aspect is the unmasking of spurious correlations utilized by the AI model winning the PASCAL VOC competition (Everingham et al., 2010). The goal of this prestigious challenge was to train models, which reliably recognize objects such as boats, trains, or horses in real-word images crawled from Flickr. Since at the time of the challenge (2007–2012) explanation techniques were not available, the participating methods were solely evaluated on the basis of average classification precision. Several years after the challenge, however, it turned out that the seemingly well-performing competition winner was classifying many images correctly, but for the wrong reason (Lapuschkin et al., 2016). More precisely, it was recognizing boats in images solely by the presence of water, trains by the presence of rail tracks, and horses by the presence of a copyright watermark, which accidentally and unnoticedly was present in many horse images in the training and test dataset. This case demonstrates that models developed in lab settings not necessarily perform well when deployed in real applications (where horses do not carry a copyright watermark). Furthermore, it shows that a profound understanding of the model and its prediction strategies is a crucial step (though not sufficient) for the development of trustworthy AI models. Another popular application domain of explanation methods are the sciences, e.g., medicine (Lauritsen et al., 2020), pharmacology (Preuer et al., 2019), meteorology (Ebert-Uphoff and Hilburn, 2020), or astronomy (Zhu et al., 2019), where explanations are used to generate novel scientific hypotheses and help gaining deeper insights into the natural processes behind the observed data.

This chapter introduces the basic concepts and methods of explainable AI, and discusses new developments in this quickly evolving research field. It will mainly focus on XAI techniques designed (but not limited) to explain the predictions of deep neural networks, i.e., models which are commonly regarded black boxes due to their nonlinearity and enormous complexity. Despite the availability of these explanation methods, practitioners still often refrain from using black-box models and prefer machine learning algorithms such as (generalized) linear models, rule-based systems, decision trees, or Random Forests. These simpler models are commonly regarded more transparent and perform very well in many applications. However, while for a long time it has been common belief that there exists a fundamental trade-off between model complexity/performance and model explainability, recent work (Lipton, 2018; Lundberg et al., 2020) argues that this is not the case, i.e., that even the results of these simpler models can be hard to interpret for humans or the interpretations can be misleading.

The rest of this chapter is organized as follows. Section 2.2 starts with the question what it actually means to explain a prediction and what properties good and meaningful explanations (and explanation methods) should fulfill. In particular, we focus on attribution methods assigning scores to every input dimension (e.g., pixel), reflecting the importance of this dimension with respect to the model's prediction. We formally introduce the concept of explanation for linear models and discuss possible extensions to nonlinear models such as deep neural networks. Throughout the chapter, we stress that the term explanation is very ambiguous and that we therefore should be always aware what we refer to when using it. We illustrate this often overlooked aspect by discussing the filter vs. pattern problem from the neuroimaging literature (Haufe et al., 2014).

Section 2.3 reviews explanation methods. In particular, we focus on XAI techniques proposed to compute explanations—in the form of attribution maps—for individual predictions of a deep classifier. These methods include perturbation-based approaches determining the relevant parts by relating the change in the output to changes in the input, surrogate-based techniques locally fitting a simple interpretable model (e.g., linear model) to the prediction, and propagation-based algorithms redistributing the model's decision in the form of relevance values from the output to the input. In addition, we briefly also discuss whole-dataset XAI analysis methods and approaches aiming to interpret the model itself rather than its predictions.

Section 2.4 discusses some of the recent developments in XAI. Since the field is very vibrant, we focus on only two directions. First, we present recent works aiming to integrate explanations into the model training in order to debug the model, improve its generalization ability, or make it more efficient by targeted pruning and quantization. Second, we present the concept of neuralization, which allows one to transfer XAI techniques developed for explaining neural network classifiers to other model types or tasks such as clustering, regression, and outlier detection.

Section 2.5 concludes the chapter with a discussion on the limitations of current XAI methods and an outlook on interesting future directions.

2.2 Concepts

The term explanation is highly ambiguous and lacks a strict and uniformly accepted (mathematical) definition. Moreover, it can have different meanings in different contexts. For instance, if used in everyday life it usually refers to an interactive and iterative process, in which one person explains something (by using words, by referring to common concepts or by demonstration) to another person. In scientific context, an explanation, e.g., of a natural phenomenon, typically consists of deductive logic, a formal description or reference to the results of conducted experiments. When mentioning explanations in the context of machine learning, we usually refer to attribution maps computed for predictions of complex ML models (see Fig. 2.1). Misunderstandings can occur if we mix up these different types of explanations, e.g., if we expect the XAI method to generate human-like or formal descriptions of model behavior. Although some progress has been recently made (e.g., Kim et al. (2018); Achtibat et al. (2022)), the computation of semantically richer (concept-based) explanations of model behavior is still an open research topic.