Item

Abstract

State-of-the-art deep learning methods achieve human-like performance on many
tasks, but make errors nevertheless. Characterizing these errors in easily
interpretable terms gives insight into whether a model is prone to making
systematic errors, but also gives a way to act and improve the model. In this
paper we propose a method that allows us to do so for arbitrary classifiers by
mining a small set of patterns that together succinctly describe the input data
that is partitioned according to correctness of prediction. We show this is an
instance of the more general label description problem, which we formulate in
terms of the Minimum Description Length principle. To discover good pattern
sets we propose the efficient and hyperparameter-free Premise algorithm, which
through an extensive set of experiments we show on both synthetic and
real-world data performs very well in practice; unlike existing solutions it
ably recovers ground truth patterns, even on highly imbalanced data over many
unique items, or where patterns are only weakly associated to labels. Through
two real-world case studies we confirm that Premise gives clear and actionable
insight into the systematic errors made by modern NLP classifiers.