Back
Chiodini_Hauswirth_2021_ACM_KoliCalling
Conference paper (in proceedings)

Wrong answers for wrong reasons : the risks of ad hoc instruments

  • Chiodini, Luca ORCID Istituto del software (SI), Facoltà di scienze informatiche, Università della Svizzera italiana, Svizzera
  • Hauswirth, Matthias ORCID Istituto del software (SI), Facoltà di scienze informatiche, Università della Svizzera italiana, Svizzera
  • 2021
Published in:
  • Koli Calling International Conference on Computing Education Research. - 2021, no. 16, p. 1-11
Computing education
Experiment
Assessment
Multiple-choice
English To evaluate novel pedagogies, approaches, and tools, Computer Science Education researchers often conduct experiments to look for differences among groups treated with different interventions. The methodological rigor of such experiments affects the soundness of the conclusions the researchers can draw. In this paper we focus on a central aspect of such experimental research: the instruments used to assess participants’ knowledge. Specifically, we study the use of ad hoc instruments and the risks due to their insufficient validation. We present a literature survey that highlights how, even though standardized instruments exist, the majority of published experiments in the last five years at major Computer Science Education conferences carries out pre/post-tests using ad hoc instruments, often with multiple-choice as question type. We demonstrate the risks of such commonly used but insufficiently validated multiple-choice instruments. We propose a richer way to analyze and assess the correctness of answers to multiple-choice questions, requiring participants to add brief explanation texts as a justification of each answer. We run an experiment and analyze the collected answers using the two approaches, with and without explanations, to show that the risk of drawing opposite conclusions from the statistical analysis is real.
Collections
Language
  • English
Classification
Computer science and technology
Notes
  • Koli Calling '21: 21st Koli Calling International Conference on Computing Education Research
  • Joensuu Finland
  • November 18, 2021
License
Rights reserved
Open access status
green
Identifiers
Persistent URL
https://n2t.net/ark:/12658/srd1334761
Statistics
Document views: 4 File downloads:
  • Chiodini_Hauswirth_2021_ACM_KoliCalling: 1