Authors: Lai Wei, Pangrui Xing, Kenny K. N. Chow, Stephen Jia Wang Status: Published, International Journal of Human–Computer Interaction (SSCI, Q1; IF 4.9)

Figure 3. Visual overview of the entire pedagogical agent (PA) gestures design procedures.

Figure 3. Visual overview of the entire pedagogical agent (PA) gestures design procedures.

Abstract

Generative AI expands opportunities for embodied agents in HCI, yet a gap persists between human-centered AI principles and practical design methods, particularly for pedagogical agents’ (PAs) co-speech gestures. Automated text-to-gesture systems lack the instructional nuance needed for effective teaching. To address this, we used a Research-through-Design approach to develop a human-centered framework that translates pedagogical intent into gesture specifications for embodied AI teachers. The framework includes four iterative stages: Preparation, which analyzes gesture patterns and instructional functions; Human PA Acting, where educators and designers rehearse gestures through performance; Embodied PA Acting, which transfers human motion to agents using video-based pose estimation; and EPA-assisted Course Delivery, which evaluates student experiences through interviews. Findings indicate that these gestures enhanced students’ perception of the PA’s professionalism, approachability, and instructional rhythm, while boosting overall engagement. This work contributes a design framework and insights for pedagogical gesture design, and exploratory guidance for generative-AI prompting.

Figure 1. Key steps in our human-centered design methodology: (A) gesture analysis: Understanding teachers’ intents by observing a senior teacher’s gestures. (B) Educator’s improvised gestures: Capturing tacit instructional knowledge through performance. (C) PA designer mirrors educator: Facilitating interdisciplinary collaboration and refinement. (D) Human-to-EPA transformation: Upper body motions extracted via DeepMotion for 3D animation. (E) EPA and teaching materials integration: 3D animation embedded alongside teaching materials.

Figure 1. Key steps in our human-centered design methodology: (A) gesture analysis: Understanding teachers’ intents by observing a senior teacher’s gestures. (B) Educator’s improvised gestures: Capturing tacit instructional knowledge through performance. (C) PA designer mirrors educator: Facilitating interdisciplinary collaboration and refinement. (D) Human-to-EPA transformation: Upper body motions extracted via DeepMotion for 3D animation. (E) EPA and teaching materials integration: 3D animation embedded alongside teaching materials.

Figure 4. Stage 1: Preparation.

Figure 4. Stage 1: Preparation.

Figure 5. Stage 2: Human PA acting.

Figure 5. Stage 2: Human PA acting.

Figure 6. Stage 3: EPA acting.

Figure 6. Stage 3: EPA acting.

Figure 7. Stage 4: EPA-assisted course delivery.

Figure 7. Stage 4: EPA-assisted course delivery.

Paper: Grounding Pedagogical Intents in Embodied AI Teachers: A Human-Centered Framework for Designing and Evaluating Instructional Gestures

DOI: 10.1080/10447318.2026.2664083