Explainable Machine Learning

PD Dr. Ullrich Köthe, Prof. Carsten Rother, SS 2018
Thursdays, 14:00-16:00, HCI, Mathematikon B (Berliner Str. 43), 3. floor, SR 128

Today's machine learning algorithms, and in particular neural networks, mostly act as blackboxes: They make very good predictions, but we don't really understand why. This is problematic for various reasons: Why should users (e.g. physicians) trust these algorithms? Will blackbox methods contribute to the advancement of science, when they produce numbers, not insight? How can one legally challenge an objectionable machine decision? Explainable machine learning attempts to solve these problems by opening the blackbox. In the seminar, we will look at many different ideas on how this can be done and what kind of insight is gained.

Since 29 students are registered for the seminar, we will have two talks every week, either on two related topics or on a single more complex topic. Please send me an email with your favourite topics, especially if you want to present at the beginning of the semester.


19. April Felix Feldmann: Experimental Investigation of Explainability Slides | Report
Conrad Sachweh: The EU "right to explanation"; An explainable machine learning challenge Slides | Report
26. April Mohammad Gharaee: Grad-CAM Slides | Report
Philipp Wimmer: Interpreting and understanding deep neural networks Slides | Report
3. May Philip Grassal: Why Should I Trust You? Slides | Report
Christoph Schaller: What is Relevant in a Text Document? Slides | Report
17. May Carsten Lüth: Dynamic routing between capsules Slides | Report
Michael Dorkenwald: Matrix capsules with EM routing Slides | Report
24. May Philipp Reiser: Metric learning with adaptive density discrimination Slides | Report
Benedikt Kersjes: Learning Deep Nearest Neighbor Representations Using Differentiable Boundary Trees Slides | Report
Jens Beyermann: Deep Unsupervised Similarity Learning using Partially Ordered Sets Slides | Report
7. June Sebastian Gruber: What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? Slides | Report
Florian Kleinicke: Learning how to explain neural networks: PatternNet and PatternAttribution Slides | Report
14. June Philipp de Sombre: Understanding Black-box Predictions via Influence Functions Slides | Report
Thorsten Wünsche: MARGIN: Uncovering Deep Neural Networks using Graph Signal Analysis Slides | Report
21. June Pingchuan Ma: Network Dissection: Quantifying Interpretability of Deep Visual Representations Slides | Report
Johannes Vogt: Feature Visualization Slides | Report
28. June Aliya Amirzhanova: Deep feature interpolation for image content changes Slides | Report
Julian Rodriquez: Distilling a Neural Network Into a Soft Decision Tree Slides | Report
5. July Daniela Schacherer: Interpreting Deep Classifier by Visual Distillation of Dark Knowledge Slides | Report
Michael Aichmüller: Generating Visual Explanations Slides | Report
Frank Gabel: Generative Adversarial Text to Image Synthesis Slides | Report
12. July Peter Huegel: InfoGAN Slides | Report
Hannes Perrot: Inferring Programs for Visual Reasoning Slides | Report
19. July Nasim Rahaman: Discovering Causal Signals in Images Slides | Report
Stefan Radev: CausalGAN Slides | Report

Topics to Choose From:

Topic 1: Black-box Attention Indicators

Attention mechanisms explain an algorithm's decision by pointing out the crucial evidence in the data. For example, when the algorithm recognizes a cat in an image, an attention mechanism will highlight the mask presumably containing the cat. If the mask is off, there is a problem, and the algorithm should not be trusted. Black-box methods achieve this without looking into the algorithm itself and thus work for any machine learning method.

Topic 2: White-box Attention Indicators

The goal is the same (pointing out the evidence), but the machine learning algorithm is opened-up and extended to facilitate the search. Often, a modified version of back-propagation is used to trace high neuron activations in a network's output layer (indicating high evidence for a class) all the way through the network to the input.

Topic 3: Feature Visualization

Feature visualization tries to make visible what a neural network has learned: What patterns cause high activations in the network's interior layers? What forms or objects do individual neurons specialize in? Is there a "grandma neuron"?

Topic 4: Influential Training Examples

A training example is influential for a particular prediction task if the prediction would change significantly had the example been missing from the training set. The result of this analysis reveals typical prior instances for the present situation, as well as typical counter-examples, thus providing an "explanation by analogy".

Topic 5: Confidence Estimation

When a system is able to provide reliable self-diagnosis and to point out cases where the results should be ignored, it becomes much more trustworthy, even when it cannot explain its reasoning.

Topic 6: Reduction of Complex Models to Simpler Ones

Powerful methods like neural networks are incomprehensible for humans. However, one can use complex models to train simpler ones for special cases (e.g. in the neighborhood of an instance of interest, or for an important subproblem), which can then be understood.

Topic 7: Text/Caption/Rule/Program Generation

Texutual descriptions and rule sets are easy to understand, so it makes sense to extract them from the implicit knowledge of a complex method. In an advanced form, neural networks perform program induction, generate simple programs that can then be run to answer queries of interest.

Topic 8: Similarity Learning

Similarity is one of the most fundamental modes of explanation. However, it is very difficult to define the similarity between complex situations or entities in a way that conforms to human intuition. Simple criteria such as Euclidean distance don't work well, and more suitable metrics are best obtained by learning.

Topic 9: Learning Disentangled Representations

Many learning methods transform the raw data into an internal representation where the task of interest is easier to solve. When these hidden variables carry semantic meaning, their activations can serve as categories for explanation. Suitable modifications of the network architecture and/or training procedure encourage these variables to disentangle into meaningful concepts.

Topic 10: Learning with Higher Level Concepts and/or Causality

Even more meaningfuls descriptions can be achieved when learned latent attributes are connected to high-level abstractions such as object hierarchies or causal graphs. Recent work has shown very promosing results in this direction.

Topic 11: Application Perspectives