English
 
Help Privacy Policy Disclaimer
  Advanced SearchBrowse

Item

ITEM ACTIONSEXPORT

Released

Conference Paper

Designing Decision Support Systems Using Counterfactual Prediction Sets

MPS-Authors
/persons/resource/persons275068

Straitouri,  Eleni
Group M. Gomez Rodriguez, Max Planck Institute for Software Systems, Max Planck Society;

/persons/resource/persons75510

Gomez Rodriguez,  Manuel       
Group M. Gomez Rodriguez, Max Planck Institute for Software Systems, Max Planck Society;

External Resource
No external resources are shared
Fulltext (restricted access)
There are currently no full texts shared for your IP range.
Fulltext (public)

arXiv:2306.03928.pdf
(Preprint), 3MB

straitouri24a.pdf
(Publisher version), 3MB

Supplementary Material (public)
There is no public supplementary material available
Citation

Straitouri, E., & Gomez Rodriguez, M. (2024). Designing Decision Support Systems Using Counterfactual Prediction Sets. In Forty-first International Conference on Machine Learning (pp. 46722-46744). PMLR. Retrieved from https://arxiv.org/abs/2306.03928.


Cite as: https://hdl.handle.net/21.11116/0000-000F-19FF-7
Abstract
Decision support systems for classification tasks are predominantly designed
to predict the value of the ground truth labels. However, since their
predictions are not perfect, these systems also need to make human experts
understand when and how to use these predictions to update their own
predictions. Unfortunately, this has been proven challenging. In this context,
it has been recently argued that an alternative type of decision support
systems may circumvent this challenge. Rather than providing a single label
prediction, these systems provide a set of label prediction values constructed
using a conformal predictor, namely a prediction set, and forcefully ask
experts to predict a label value from the prediction set. However, the design
and evaluation of these systems have so far relied on stylized expert models,
questioning their promise. In this paper, we revisit the design of this type of
systems from the perspective of online learning and develop a methodology that
does not require, nor assumes, an expert model. Our methodology leverages the
nested structure of the prediction sets provided by any conformal predictor and
a natural counterfactual monotonicity assumption to achieve an exponential
improvement in regret in comparison to vanilla bandit algorithms. We conduct a
large-scale human subject study ($n = 2{,}751$) to compare our methodology to
several competitive baselines. The results show that, for decision support
systems based on prediction sets, limiting experts' level of agency leads to
greater performance than allowing experts to always exercise their own agency.
We have made available the data gathered in our human subject study as well as
an open source implementation of our system at
https://github.com/Networks-Learning/counterfactual-prediction-sets.