| Author: | Wei, Lai |
| Title: | Animated gestures : pedagogical agents’ meaningful instructional gesture design for effective educational information presentation |
| Advisors: | Chow, K. N. Kenny (SD) |
| Degree: | Ph.D. |
| Year: | 2024 |
| Subject: | Artificial intelligence -- Educational applications Gesture Communication and education Computer-assisted instruction Hong Kong Polytechnic University -- Dissertations |
| Department: | School of Design |
| Pages: | xiv, 206, 28 pages : color illustrations |
| Language: | English |
| Abstract: | Agents (e.g., virtual agents, intelligent devices, robots) are becoming increasingly competent at collaborating closely with humans in various contexts, benefiting mutually from such collaborations due to advances in artificial intelligence (AI) technologies. Meanwhile, the COVID-19 pandemic has accelerated the development of remote communication technology. Various industries have exhibited a rising demand for comprehensive and precise information presentation. In the context of online education, research on multimodal information presentation by Pedagogical Agents (PAs) has been discussed for decades. As a visual component in multimedia learning environments, PAs play an essential role in fostering a sense of social involvement and promoting students' learning engagement and performance. Scholars have investigated the features of PAs, such as appearance, facial expressions, nonverbal behaviours, and interactions. They determined that the design of PAs should be modelled on human lecturers' natural verbal and nonverbal behaviours. Based on McNeill (2011) gesture classification, several educational researchers have reported that five types of PA gestures (i.e., cohesives, beats, deictics, iconics, and metaphorics) show promise in benefiting students' learning experiences. These benefits include capturing students' visual attention, eliciting students' emotional reactions, and promoting students' social engagement and learning performance. Subsequently, computer science researchers have intensively investigated co-speech gesture creation, achieving high accuracy in generating gestures that coincide with speech features, such as prosody, rhythm, and semantic meaning of words. However, students' comprehension and mental effort regarding information acquisition through lecturers' gestures in real-life lecture settings remain underexplored. As a result, the student-centered design of PA's meaningful instructional gestures is still an uncharted research area. The study of natural gesture and speech alignment in human lecturers can provide valuable insights for the development of PAs equipped with meaningful instructional gestures. However, the consensus on this alignment between lecturers and students has not been extensively explored. Investigating the in-class perceptions and understanding of gestures by both lecturers and students is essential for the design of student-centered PA gestures. This study aims to guide the design of PAs by focusing on instructional gestures that are meaningful, as determined by the in-class perceptions and agreed-upon gesture-speech alignment of both lecturers and students. The implementation of meaningful instructional gestures in PAs has the potential to significantly enhance students' learning experiences and outcomes. The findings of this study could be instrumental in developing PAs that generate gestures synchronized with speech, thereby enriching virtual learning experiences. From a practical standpoint, this research contributes to the advancement of design guidelines for gestural PAs, supporting instructional designers in creating more effective multimedia learning environments. A series of three studies were conducted using Multimedia Learning Theory (MLT; Mayer, 2002) and Cognitive Load Theory (CLT; Sweller, 2011) as foundational theoretical frameworks. The first study employed an eye-tracking experiment to assess students' learning performance in a virtual learning environment. It specifically examined the correlations between students' eye movements and the visual components of the interface, namely virtual agents and chatboxes, across four types of conventional interface layouts. The results supported the principles of MLT and CLT, suggesting that multimedia learning can enhance students' retention scores and reduce cognitive load. A subsequent study involved a scoping systematic review that focused on the impact of Pedagogical Agent gestures on students' learning experiences. This review revealed that PAs equipped with meaningful gestures have a positive influence on students' learning experiences and performance. The findings from the initial study provided a theoretical foundation for the subsequent investigation in Study Two. In this study, I analyzed gesture-word alignment levels and developed categories for meaningful instructional gestures. This involved observing real-life classroom interactions between lecturers and students, complemented by comprehensive interviews. Additionally, I applied McNeill's gesture classification and Natural Language Processing (NLP) to identify synchrony between words and gestures in human lecturers. Drawing on these insights, I proposed a design paradigm centered on word-gesture alignment patterns, aiming to facilitate the creation of PAs with effective instructional gestures. Study Three then progressed to executing two between-subject experiments. These experiments were designed to examine the impact of three-dimensional (3D) animated PAs, which incorporated meaningful instructional gestures, on students' learning experiences and outcomes. This advanced research suggests that animated PAs with meaningful instructional gestures can effectively enhance students' learning outcomes and experiences. This research explored the features and and impacts of multimodality in lecture discourse across various learning environments, including online, offline, and hybrid settings. It offered an in-depth analysis of both lecturers' and students' abilities to comprehend multimodal information, along with microanalyses of gesture and part-of-speech alignment in lecture contexts. The study's categorization of meaningful instructional gestures, which integrates McNeill's gesture classification, presents a student-centered design framework for gestural PAs. By interacting with PAs featuring four types of meaningful instructional gestures, students' comprehension of new knowledge and overall learning experiences can be significantly enhanced. The proposed design model for the PA's meaningful instructional gestures demonstrates potential for effective presentation of educational information. |
| Rights: | All rights reserved |
| Access: | open access |
Copyright Undertaking
As a bona fide Library user, I declare that:
- I will abide by the rules and legal ordinances governing copyright regarding the use of the Database.
- I will use the Database for the purpose of my research or private study only and not for circulation or further reproduction or any other purpose.
- I agree to indemnify and hold the University harmless from and against any loss, damage, cost, liability or expenses arising from copyright infringement or unauthorized usage.
By downloading any item(s) listed above, you acknowledge that you have read and understood the copyright undertaking as stated above, and agree to be bound by all of its terms.
Please use this identifier to cite or link to this item:
https://theses.lib.polyu.edu.hk/handle/200/12889