Notes on : ‘Mastering knowledge: the impact of generative AI on student learning outcomes’
generative AI
learning outcomes
qualitative analysis
Source
Pallant, J. L., Blijlevens, J., Campbell, A., & Jopp, R. (2025). Mastering knowledge: the impact of generative AI on student learning outcomes. Studies in Higher Education, 1–22
Summary
- Study population:
- The study involved students studying marketing in Melbourne, Australia.
- Research question(s): This paper had two primary objectives:
- To explore whether students’ use of generative AI (GenAI) is influenced by their goal orientations, and
- To assess how students’ use of GenAI impacts their learning outcomes
- Methods:
- A quasi-experimental approach was adopted, as ethical considerations prevented exposing one group to GenAI while withholding it from another.
- In Week 1, students used GenAI to generate a definition of a concept relevant to their course. After completing the 12-week unit, they wrote their own definition, compared it with the AI-generated one, and reflected on their learning.
- Approximately 75,000 words of student responses were analyzed using quantitative content analysis (QCA), which combined a qualitative coding framework with quantitative analyses applied after coding.
- The coding framework captured broad themes [and more specific codes]:
- Whether GenAI use aligned with a mastery goal structure [constructive approach or augmenting knowledge] or a performance goal structure [regurgitating knowledge or procedural approach]
- Types of learning outcomes [information literacy,…]
- Types of thinking capability [applied knowledge, critical thinking,…]
- Final unit marks (%) and assignment marks (%) were used as indicators of student learning, both for the GenAI task and overall unit performance.
- Selected results:
- In total, 198 students from three universities were enrolled in the study: 56 and 84 students in two undergraduate units, and 58 in a postgraduate unit.
- Six students did not complete the study, resulting in a final sample of 192 students included in the analysis.
- Students who used a constructive approach scored higher on both the assignment and the overall unit compared to those who did not (mean differences of 6.2%, p < 0.05, and 5.5%, p < 0.05, respectively).
- Students who used a procedural approach scored lower on both the assignment and the overall unit compared to those who did not (mean differences of –6.2%, p < 0.05, and –5.6%, p < 0.05, respectively).
- Selected practical implications:
- Design assessments that ask students to compare and contrast GenAI responses with their own.
- Encourage students to reflect on how they collaborated with GenAI to reach their answers.
- Such assessments help students connect GenAI output to course content, develop reasoning skills, and recognize how GenAI (as a more knowledgeable source) can support learning within their zone of proximal development.
Key Quotes
“The unique and complex landscape of GenAI. along with its potential impact on the student experience and implications for learning outcomes, remains largely unknown.”
“By understanding the mechanisms behind students’ attitudes toward and use of GenAI in their learning will provide clearer guidance on how to tackle concerns and leverage opportunities”
Reflection
- It is unclear what makes this study a quasi-experiment, as it is not evident what the treatment or control groups would have been in a true experimental design.
- Using grades as a measure of learning in a research setting has limitations. It would strengthen the study if the authors incorporated established assessment instruments or published questions and rubrics that are specifically designed to evaluate learning outcomes.
- Assignment grades are likely based on how well students critically evaluated differences between their own definition and the AI-generated one (the rubric was not provided). Given this, it’s unsurprising that students classified as having a mastery goal orientation received higher marks. The students are grouped into goal orientation categories based on their written responses and those same responses are also used to determine their grades. It’s unclear whether students with a certain goal orientation tend to perform better, or whether their higher performance simply led to them being placed in that category. This makes it hard to draw any clear cause-and-effect conclusions. Therefore, based on my understanding, this study design is not able to assess the impact of GenAI on learning outcomes (study objective 2).
Relevant Background
- Achievement Goals Framework is a well-established theory in education research, often used to study academic behaviours such as cheating and to explain how positive learning outcomes are achieved. The framework posits that students adopt either performance goals and/or mastery goals in their approach to learning. Performance goal orientations emphasize demonstrating competence relative to others, focusing on grades, comparison, and external validation. While this mindset can offer short-term motivation, it often undermines deep learning and discourages students from taking intellectual risks. In contrast, mastery goal orientations prioritize the development of competence and a genuine pursuit of understanding. Students with this mindset tend to embrace challenges as opportunities for growth, leading to more sustained and meaningful learning. This paper proposes that similar patterns may emerge in how students integrate generative AI into their learning, depending on their underlying goal orientation.
- Zone of Proximal Development (ZPD) describes the difference between what a learner can accomplish independently and what they can achieve with guidance from a more knowledgeable person. As learners gain experience and skills, their ZPD evolves. Generative AI (GenAI) can offer tailored support within a student’s current ZPD and adjust as the learner progresses. This approach aligns with constructivist pedagogy, which emphasizes authentic, collaborative learning that challenges students within their ZPD.