hide
Free keywords:
-
Abstract:
Research on decentralized systems such as peer-to-peer overlays and ad hoc
networks has been hampered by the fact that few systems of this type are in
production use, and the space of possible applications is still poorly
understood. As a consequence, new ideas have mostly been evaluated using common
synthetic workloads, traces from a few existing systems, testbeds like
PlanetLab, and simulators like ns-2. Some of these methods have, in fact,
become the gold standard for evaluating new systems, and are often a
prerequisite for getting papers accepted at top conferences in the field.
In this paper, we examine the current practice of evaluating decentralized
systems under these specific sets of conditions and point out pitfalls
associated with this practice. In particular, we argue that (i) despite
authors' best intentions, results from such evaluations often end up being
inappropriately generalized; (ii) there is an incentive not to deviate from the
accepted standard of evaluation, even if that is technically appropriate; (iii)
research may gravitate towards systems that are feasible and perform well when
evaluated in the accepted environments; and, (iv) in the worst case, research
may become ossified as a result. We close with a call to action for the
community to develop tools, data, and best practices that allow systems to be
evaluated across a space of workloads and environments.
