Back
2023INF021
Doctoral thesis

Studying strengths and weaknesses of code recommenders

  • 2023

PhD: Università della Svizzera italiana

Code completion
Recommender system
Deep learning
Empirical software engineering
English Given the high costs of software development and maintenance, tools and techniques have been proposed both in industry and academia to speed up programming activities. Some of these techniques focus on automating the generation of source code, by predicting the code tokens the developer would write starting from the ones already typed. We use the term code recommender to refer to these approaches. In this thesis, we study the strengths and the weaknesses of code recommenders from several perspectives. We started by investigating the accuracy of recently proposed Deep Learning (DL) models in generating non-trivial and possibly multiple code statements. Indeed, previous work usually tested these techniques in the relatively easy task of predicting the single next token the developer is likely to write. We pushed the boundaries of the predictions to entire blocks of code, showing that DL models can achieve excellent results when predicting a few code tokens but their accuracy steadily decreases when asked to predict entire code statements. Still, even in such a challenging scenario, DL models are able to produce correct suggestions in ∼29% of cases. Given such a finding and the stunning capabilities of recently released products such as GitHub Copilot, we decided to investigate the extent to which DL models tend to copy code from the training set. Surprisingly, we found that DL models rarely copy verbatim from the training data, mostly generating original code. Finally, we studied the extent to which DL models generalize across different versions of the same programming language, showing that a model trained on a language version vi exhibits a strong drop in performance when asked to predict completions on a version vj != vi. Then, since most of the tools proposed in the literature are based on researchers’ intuitions, we ran a survey with 80 practitioners to create a taxonomy of characteristics of code recommenders they consider important (e.g., adapt the recommendations to the developer’s programming style). Such a taxonomy can be used to inform future research in the field aimed at targeting the weaknesses of code recommenders as perceived by practitioners. To show that, we tackle one of the identified weaknesses (i.e., limited knowledge of the coding context) and show how the performance of code recommenders can be boosted by enriching the coding context they are aware of before triggering a recommendation.
Collections
Language
  • English
Classification
Computer science and technology
License
License undefined
Open access status
green
Identifiers
Persistent URL
https://n2t.net/ark:/12658/srd1327659
Statistics
Document views: 75 File downloads:
  • 2023INF021.pdf: 154