Back
2022INF004
Doctoral thesis

Automatic generation of test oracles from natural language specifications

  • 2022

PhD: Università della Svizzera italiana

Testing
Oracles
Natural language
English This thesis proposes a framework to automatically derive test oracles from natural language specifications. We studied and developed cost-effective techniques to derive oracles from information commonly available in natural language about the code. Despite previous research in software testing, the oracle problem, that is, the challenge to distinguish correct from incorrect behavior, is still largely open. Contemporary test case generators rely on either simple and incomplete implicit oracles or on regression oracles that refer to the results of running previous versions of the program. Implicit oracles can reveal exceptions and program crashes, but miss semantically relevant issues. Regression oracles can detect deviations from the behavior observed in former versions of the program, but not in new functionalities. Many approaches generate powerful and complete test oracles from formal specifications that are still not the most common practice in software development. The main goal of this thesis is to define techniques that use natural language annotations, which current approaches largely ignore, to generate effective test oracles without additional human effort. Most software systems are supported by textual information, such as annotations, comments, and wikis. This information is typically informal and unstructured, and often combines natural language expressions with developers’ jargon. We also observe that informal artifacts are prone to human mistakes. In a nutshell, informal artifacts are not always reliable and are hard to exploit automatically. This thesis defines approaches to automatically interpret and translate informal and unstructured information that combines natural language expressions with developers’ jargon into actionable test oracles, while overcoming human errors that affect their quality. We first automatically verify the consistency between the code and its documentation, and signal developers’ inconsistencies that we detect at a fine-grained level. We then process the pruned specifications to automatically generate test oracles in the form of executable assertions. We designed and developed approaches that process both structured and unstructured Javadoc specifications, to derive both descriptive and prescriptive assertions for the methods of a Java class. We successfully experimented with the approaches on popular, widely-used and open-source Java systems. This thesis opens new research directions towards the automated exploitation of natural language artifacts beyond Javadoc specification, to derive powerful test oracles. The intuitions and observations our study proposes can be generalized and applied to many other software related artifacts.
Collections
Language
  • English
Classification
Computer science and technology
License
License undefined
Open access status
green
Identifiers
Persistent URL
https://n2t.net/ark:/12658/srd1321073
Statistics
Document views: 47 File downloads:
  • 2022INF004.pdf: 45