Should we Call Annie? - A mixed-methods intervention study examining the impact of one chatbot on the vocabulary acquisition of upper secondary students in Sweden

Abstract

Aim The primary aim of this thesis is to investigate whether the use of a generative AI chatbot – the embodied conversational agent Call Annie – has a significant (positive) effect on Swedish upper secondary students' contextual vocabulary acquisition over time, compared to a more traditional instructional approach. In addition, the study explores potential correlations between quantitative outcomes and qualitative data derived from mid-intervention observations and a post-intervention survey on attitudes toward the chatbot. Enhanced fluency and pronunciation are expected to be perceived byproducts of the verbal interaction with Call Annie. Theory The theoretical framework of the thesis is anchored in Communicative Language Teaching (CLT) and Task-based Language Teaching (TBLT), with specifically designed intervention tasks being employed to promote vocabulary acquisition. In order to facilitate the process, Call Annie is prompted to engage in negotiation of meaning, assuming the role as a more knowledgeable other person, subsequently providing the students with comprehensible input via meaningful conversations. Method The thesis utilizes a mixed-methods design, which consists of quantitative evaluation research and qualitative explanatory research. During an intervention, two groups are subjected to five tasks and five tests, with a subsequent delayed post-test. While the experimental group (n = 31) are instructed to solve the tasks by interacting with Call Annie, the instructional method in the control group (n = 27) is more traditional. In an effort to find potential correlations, observations made during the intervention are eventually compared to both the statistical results and the participants' answers to a brief survey, via a reflexive thematic analysis. Results Even though both groups had a significant increase in vocabulary scores, p < .001, from pre-test to delayed post-test, an ANCOVA revealed no significant difference between the experimental and control groups, F(1, 35) = 1.35, p = .25, suggesting that the instructional method accounted for a small proportion of the variance in delayed post-test scores. The findings indicate that AI-supported learning through Call Annie is at least equivalent to traditional instruction, suggesting its viability as an alternative pedagogical approach rather than a superior or inferior one.