SOFTWARE SYSTEMS SUMMER SCHOOL
The Truth, the Whole Truth, and Nothing but the Truth: A Pragmatic Approach to Assessing Empirical Evaluations
An unsound claim can misdirect a field, encouraging the pursuit
of unworthy ideas and the abandonment of promising ideas. An
inadequate description of a claim can make it difficult to reason
about the claim, for example to determine whether the claim is
sound. Many practitioners will acknowledge the threat of
unsound claims or inadequate descriptions of claims to their
field. We believe that this situation is exacerbated and
even encouraged by the lack of a systematic approach to
exploring, exposing, and addressing the source of unsound claims
and poor exposition.
This talk discusses a framework that identifies three “sins" of reasoning that lead to unsound claims and two “sins" of exposition that lead to poorly described claims. Sins of exposition obfuscate the objective of determining whether or not a claim is sound, while sins of reasoning lead directly to unsound claims.
Our framework provides practitioners with a principled way of critiquing the integrity of their own work and the work of others. We hope that this will help individuals conduct better science and encourage a cultural shift in our research community to identify and promulgate sound claims.
Steve Blackburn is a professor in the Research School of Computer Science at the Australian National University. His research interests include programming language implementation, architecture, and performance analysis. Steve is heavily involved in two major research infrastructure projects; the DaCapo benchmark suite and Jikes RVM.