Stakeholder-Driven Contextual Evaluation of Language Models in Education
About
With the increasing reliance of AIED on opaque, black-box scaffolds such as large language models to support student learning, there is a growing concern about their limitations when used in diverse pedagogical contexts. This opacity often undermines stakeholders' trust and shapes their perceptions, contributing to resistance toward the adoption of AI scaffolds in schools.
To address these challenges, we developed AIBAT, a workflow and system designed to support stakeholders in auditing and critically evaluating the potential benefits and harms of AI systems within their specific pedagogical contexts (e.g., subject matter, grade level, English proficiency). With AIBAT, stakeholders can specify expected behaviors — i.e., what they anticipate the AI scaffold should do — and test the system against those expectations.
In this half-day tutorial, participants will use AIBAT to identify and make sense of AI-related risks and use evidence to calibrate their trust in AI scaffolds. At the end of the tutorial, we will deliberate on AI auditing processes and discuss broader implications for promoting responsible and effective stakeholder participation in the evaluation and deployment of AI systems in educational settings.
Call for Participation
We invite 20–30 educational AI stakeholders — including researchers, teachers, developers, and practitioners — participating in AIED, EDM, or L@S to join this half-day tutorial.
Participants will explore a new evaluation method (behavior analysis) and a corresponding tool (AIBAT) to learn how they can analyze AI system behaviors to contextually evaluate AI and identify the benefits and harms of using AI with diverse student data.
No prior experience with AI auditing or NLP is required. Participants are encouraged to come with a use case or AI tool from their own context that they would like to evaluate.
Topics
- Stakeholder-driven evaluation of large language models
- Contextual AI auditing in K–12 and higher education
- Behavior analysis for equitable AI in classrooms
- Responsible AI deployment in educational settings
- Human–AI trust and transparency
- Linguistic variation and fairness in AI grading models
Important Dates
- Submission DeadlineTBD
- Notification of AcceptanceTBD
- Tutorial DateTBD
Submission Guidelines
To participate in this tutorial, please submit a short statement of interest via EasyChair. Submissions will be reviewed by the organizers to ensure a productive and focused group experience.
Submission Types
Statement of Interest
A 1–2 paragraph statement describing your background, the AI system or context you work with, and what you hope to gain from the tutorial.
Position Paper (Optional)
A short paper (2–4 pages) describing a challenge, use case, or perspective related to stakeholder-driven evaluation of AI in education. Accepted papers will be shared with tutorial participants.
Format Requirements
- Submissions should be written in English.
- Use the AIED 2026 paper template (Springer LNCS format).
- Submit via EasyChair using the link below.
Schedule
Opening Remarks
Discussion on the evaluation crisis with LLMs in educational settings.
Small Group Deliberation
Share-out on current approaches to AI evaluation across participants' contexts.
Introduction to Behavior Analysis
Overview of the behavior analysis framework underpinning AIBAT.
Individual Think Time
Participants identify relevant linguistic variations in their own use cases.
Whole Group Demo of AIBAT
Live walkthrough of AIBAT using a participant-contributed use case.
Break
Small Group Work with AIBAT
Hands-on auditing session where participants evaluate an AI system using AIBAT.
Fishbowl Demo
Small group shares findings with the whole group in a fishbowl format.
Closing Reflection
Discussion on effective stakeholder participation in the evaluation and deployment of AI.
Related Paper
AIBAT: AI Behavior Analysis Tool for Teacher-Driven Contextual Evaluation of Language Models in Education
26th International Conference on Artificial Intelligence in Education (AIED) · 2025
View Paper
Organizers
University of Wisconsin–Madison
University of Wisconsin–Madison
University of Wisconsin–Madison