Artificial Intelligence (AI) is making more decisions for us than ever before. AI is helping us keep our cars on the right lane, helping judges make the right decision, and even deciding who lives or dies on the battlefield.

As AI proliferates in our daily lives, there is also a growing fear that humans lose control. The European Commission's current president, Ursula Von der Leyen, has pushed hard to start creating frameworks to regulate the use of AI, resulting in a document guidelining the requirements that AI systems need to meet in order to be trustworthy. One of the key elements of these guidelines is the notion that "AI systems and their decisions should be explained in a manner adapted to the stakeholder concerned." What the European Commission really wants then, is Explainable AI (EAI): an AI where the logic for making the decision, or a summary of the logic is made available.

It's hard to argue with the intention behind this particular element of the guidelines. Many AI algorithms are really black boxes: partially, or not understood both by those who create them and those who interact with them. Obviously, this is problematic: there are risks both for the companies and organizations that deploy these AIs, and the people who interact with them. More explainable AIs seem to be in everyone's best interests. Nevertheless, good intentions and practice often clash: there are real, pragmatic reasons why many AIs are not engineered in such a way that they are easily explained.

What causes some AI to not be explainable?

To understand what makes some AIs hard to explain, let's start with an example of an algorithm that is easy to explain: the Body Mass Index (BMI). BMI is an algorithm that classifies people into weight groups, such as underweight, normal weight, overweight, etc. The algorithm is easy to explain: take your weight in kilograms (for example 80kg) and divide it by the square of your height in meters (e.g 1.80m times 1.80m) to come up with your BMI (in our case: 80/(1.80*1.80) = 24.6). If this number is below 20, you're likely underweight, and if it's above 25, you're likely overweight. This algorithm is easy to understand for a variety of reasons: the input data in our algorithm, your weight and height, are measured using well-defined conventions (kilograms and meters, in this case). The operation of the algorithm is also relatively simple: a division and a multiplication. Finally, the output of the algorithm is easy to interpret. While the BMI's use has been extensively criticized, you can't fault its explainability.

Let's take an example of an algorithm that is harder to explain: imagine that we'd need to create a machine learning algorithm that predicts a customer's personality profile based on their social media feed. Clearly, this problem is challenging. The data we would feed to an algorithm would be way less standardized than kilograms and meters: it would text in multiple languages, sometimes including strange characters like hashtags and references to other users. It would contain pictures and videos, links to other social media posts, and much more. We'd probably need to preprocess the data we used, and have specialized algorithms for each type of data: an algorithm to detect the language of the post, another algorithm to treat English text, ... The operations that these algorithms would need to execute would be much more complex than a multiplication and addition - modern neural networks, a type of machine learning algorithm, make millions, if not hundreds of millions of operations on a single picture or piece of text. Finally, the output of our algorithm is complex: measuring personality is hard and open to discussion and interpretation. As you can imagine, having an engineer explain why a machine learning algorithm has classified someone as being "neurotic", based on your social media post, can quickly become a very abstract and technical discussion.

The underlying reasons why AI can be hard to explain

We believe there are 8 underlying reasons why an AI solution can become hard or impossible to explain.

Reason 1: The way data is generated is not understood

The base resource that machine learning engineers work with is data. However, the exact meaning and source of this data is often nebulous and prone to misinterpretation. Data might come from a CRM, be self-reported and collected through a survey, purchased from a third-party provider, ... To make matters worse, machine learning engineers often only have a label to work with, and no further details. For example, we could have a dataset that contains a user for each row, and one column named post_count. A seasoned machine learning engineer will immediately start asking questions: count of posts since when? Does this include deleted posts? What is the exact definition of a post? Sadly, while answering this for a single column is often doable (but resource-intensive), answering it for thousands of columns is both extremely time-consuming and complex.

This brings us to our second underlying reason...

Reason 2: The data given to an algorithm is feature-rich

In a quest to have more predictive power, and thanks to the ever-growing computational power of our computers, most machine learning practitioners tend to work with very large, very feature rich datasets. With feature-rich, we mean that for every observation (e.g. a person whose personality we want to predict, our row in our previous example), we have many different types of data (e.g. timestamped posts, their interactions with other users, their signup date, ..., our columns in our previous example). It's quite common to have thousands (and many, many more) different types of data in many machine learning problems.

Reason 3: The way data is processed is complex

Machine learning engineers often don't just take the data as such and feed it to an algorithm, the process it ahead of time. Data can be enriched (creating additional data types from existing ones: such as turning a date into another variable that says if it's a national holiday or not), combined (such as reducing the output of many sensors to just a few signals) and linked (by getting data from other data sources). Each of these operations bring additional complexity in understanding and explaining the base data the algorithm is learning from.

Reason 4: The way additional training data is generated (augmentation) is complex

Many use cases of machine learning allow for the generation of additional training data, called augmentation. Homever, these generative approaches to getting more and better training data can often be complex and modify the learnings of the algorithm in subtle, unintuitive ways.

Reason 5: The algorithms that are used don't balance complexity and explanatory power (regularization)

It's often difficult to balance the predictive explanatory power of a model and the complexity of a model. Luckily, there are a slew of techniques available today to do just that for machine learning engineers, called "regularization" techniques. These techniques weigh the cost of adding complexity versus the additional explanatory power that this complexity bring, and attempt to strike a good balance. The under- or mis-application of regularization in models can lead to very, very complex models.

Reason 6: The algorithms that are used are allowed to learn unintuitive relationships (non-linearity)

Linear relationships are ones where an increase in one variable causes a set increase (or decrease) in another variable. For example, the relationship between signups to a new service and profits could be linear: for every new signup, your profit increases by a set amount. Some machine learning models can only learn linear relationships (such as the aptly named "linear regression"). These models tend to be easier to explain, but also miss out on a lot of nuance. For example, your profits might initially increase with every signup, but then decrease after a certain number of signups, because you need additional support staff for your service. While some models can learn these relationships, they are often much trickier to explain.

Reason 7: The algorithms that are used are combined (ensembling)

Many complex AI applications don't rely on a single algorithm, but a whole host of algorithms. This "chaining" of algorithms is called "ensembling". This practice is extremely common in machine learning today, but adds complexity: if a single algorithm is hard to explain, imagine having to explain the combined output of 50-100 algorithms working together.

Reason 8: There is no additional explanatory layer used

Rather than trying to make models explainable through simplicity (and as such, often sacrificing the explanatory power), another approach has emerged in the last couple of years that aim to add a glass layer on top of black-box models, that figuratively allow to peer inside the models. These models, such as Shapely Additive Explanations (SHAP), use both the data and the black-box model to explain the prediction generated by the model in question.

However, these techniques are still relatively new, and largely underused in the data science community.

When is Explainable Artificial Intelligence important (and when not)?

Clearly, creating an EAI is not trivial. While there are some best practices that should almost universally be applied (having a strong grasp of the data generation process, using regularization, and using an explanatory layer if the model becomes complex), other practices come at a cost of explainability in return for better predictive power.

Fundamentally, the higher the legal, commercial, and ethical risks of a mistake of an AI application, the more explainability (and audibility) of an algorithm becomes important.

If you're interested in finding out more on how we create EAI, please feel free to get in touch!