Back
Lanza_2023_Elsevier_JSS
Journal article

A comprehensive evaluation of SZZ variants through a developer-informed oracle

  • Rosa, Giovanni ORCID STAKE Lab, University of Molise, Italy
  • Pascarella, Luca Istituto del software (SI), Facoltà di scienze informatiche, Università della Svizzera italiana, Svizzera
  • Scalabrino, Simone STAKE Lab, University of Molise, Italy
  • Tufano, Rosalia ORCID Istituto del software (SI), Facoltà di scienze informatiche, Università della Svizzera italiana, Svizzera
  • Bavota, Gabriele Istituto del software (SI), Facoltà di scienze informatiche, Università della Svizzera italiana, Svizzera
  • Lanza, Michele ORCID Istituto del software (SI), Facoltà di scienze informatiche, Università della Svizzera italiana, Svizzera
  • Oliveto, Rocco STAKE Lab, University of Molise, Italy
Show more…
  • 2023
Published in:
  • Journal of systems and software. - 2023, vol. 202, p. 111729
SZZ
Defect prediction
Empirical study
English Automatically linking bug-fixing changes to bug-inducing ones (BICs) is one of the key data-extraction steps behind several empirical studies in software engineering. The SZZ algorithm is the de facto standard to achieve this goal, with several improvements proposed over time. Evaluating the performance of SZZ implementations is, however, far from trivial. In previous works, researchers (i) manually assessed whether the BICs identified by the SZZ implementation were correct or not, or (ii) defined oracles in which they manually determined BICs from bug-fixing commits. However, ideally, the original developers should be involved in defining a labeled dataset to evaluate SZZ implementations. We propose a methodology to define a “developer-informed” oracle for evaluating SZZ implementations, without requiring a manual inspection from the original developers. We use Natural Language Processing (NLP) to identify bug-fixing commits in which developers explicitly reference the commit(s) that introduced the fixed bug. We use the built oracle to extensively evaluate existing SZZ variants defined in the literature. We also introduce and evaluate two new variants aimed at addressing two weaknesses we observed in state-of-the-art implementations (i.e., processing added lines and handling of revert commits).
Collections
Language
  • English
Classification
Computer science and technology
License
CC BY-NC-ND
Open access status
green
Identifiers
Persistent URL
https://n2t.net/ark:/12658/srd1329460
Statistics
Document views: 16 File downloads:
  • Lanza_2023_Elsevier_JSS.pdf: 64