← Back to homepage

Standing on the Shoulders of Decaying Giants

Alexander Boll, Samuel Utzinger, Roman Bögli, Timo Kehrer

Preprint 2026

Abstract

Software engineering has improved reproducibility through replication packages that let others rerun tools and reproduce results. A replication package, however, is a snapshot, and its value as a baseline can diminish in three ways: the tool may stop working as intended, its approach may fit today’s world worse, or the evaluation setup may make comparing against it unsound. We unify these facets as tool decay. Left unchecked, tool decay means that outperforming an older tool need not demonstrate genuine progress. We study tool decay in merge conflict resolution, a long-established research area, as a first case study. We combine two complementary lines of evidence: (i) A citation analysis shows that comparisons increasingly copy published baseline numbers instead of rerunning tools, so decay goes unmeasured, and that chains of superiority claims often break. (ii) Two experiments demonstrate decay in five of eight merge tools and show that a simple method of undoing decay overshoots. Together, these findings suggest an erosion of comparability across studies: a systemic threat to the progress of software engineering research. To counter tool decay, we call for establishing shared practices to anticipate and document decay, to undo it systematically, and to sustain fair comparisons.