Authentic AI Assessment at Scale - Oxymoron or Possible?

Abstract

This presentation showcased a new AI assessment tool developed by Participate Inc. in collaboration with the University of Missouri St. Louis.  

Providing relevant, meaningful professional learning for educators is a significant and persistent challenge, particularly in under-resourced communities. Too often, professional development is standardized, disconnected from classroom realities, and fails to recognize the diverse and dynamic nature of teaching practice. This gap is especially stark for teachers serving historically marginalized communities, who face complex, real-world challenges requiring adaptive solutions rather than one-size-fits-all content. When teachers lack access to meaningful, reflective professional learning, their ability to innovate, and tailor instruction is severely limited, further exacerbating educational inequities. 

The AI assessment tool developed by Participate and the University of Missouri St. Louis was designed to address the difficulty of implementing authentic, rubric-aligned assessment within professional learning experiences that emphasize action research and reflective practice. Traditional assessments in these settings often fall short of capturing the depth and nuance of practice-based learning. Authentic assessment requires evaluating educators' ability to critically reflect, apply insights, and adapt based on real-world challenges—analyzing complex artifacts such as written reflections, project documentation, and iterative action research reports. Scaling this kind of nuanced evaluation typically relies on extensive human review, which is labor-intensive, subjective, and difficult to implement consistently across large cohorts.  

The AI assessment and feedback tool directly addresses this challenge. By leveraging advanced AI models and a rubric-aligned workflow, it analyzes text-based learning artifacts—such as reflective journals and action research reports—offering structured, formative feedback aligned with educators' professional learning goals. A key feature is the pre-submission feedback loop, which allows teachers to receive actionable, rubric-based guidance before final submission, fostering a culture of continuous improvement and self-assessment.  

At the core of the tool are digital badges that recognize and validate teachers' skills and growth. Each badge is tied to clear criteria articulating what teachers must demonstrate in their practice. Educators submit a range of artifacts—lesson plans, student work samples, and reflective narratives—to earn these badges. Rubrics are embedded directly into the badge framework, providing a transparent and consistent lens for evaluation that remains adaptable to individual classroom contexts. 

By automating significant portions of the review process, the tool reduces the manual assessment burden on human mentors, freeing them to focus on higher-order mentorship, targeted support, and meaningful collaboration. This creates a sustainable, cost-effective model for professional growth—one that ensures teachers in geographically dispersed or resource-limited contexts have equitable access to timely, consistent, and meaningful feedback. 

The presentation provided an overview of the tool and invited critical audience engagement, with particular interest in exploring the ethical considerations and potential pitfalls of AI-driven assessment in professional learning contexts. 

Keywords: artificial intelligence, digital badges, professional learning, formative feedback