hide
Free keywords:
cs.SI
Abstract:
A wide variety of online platforms use digital badges to encourage users to
take certain types of desirable actions. However, despite their growing
popularity, their causal effect on users' behavior is not well understood. This
is partly due to the lack of counterfactual data and the myriad of complex
factors that influence users' behavior over time. As a consequence, their
design and deployment lacks general principles.
In this paper, we focus on first-time badges, which are awarded after a user
takes a particular type of action for the first time, and study their causal
effect by harnessing the delayed introduction of several badges in a popular
Q&A website. In doing so, we introduce a novel causal inference framework for
badges whose main technical innovations are a robust survival-based hypothesis
testing procedure, which controls for the utility heterogeneity across users,
and a bootstrap difference-in-differences method, which controls for the random
fluctuations in users' behavior over time. We find that first-time badges steer
users' behavior if the utility a user obtains from taking the corresponding
action is sufficiently low, otherwise, the badge does not have a significant
effect. Moreover, for badges that successfully steered user behavior, we
perform a counterfactual analysis and show that they significantly improved the
functioning of the site at a community level.
