Opening the black box: exploring automated speaking evaluation

Nahal Khabbazbashi
,
Jing Xu
,
Evelina D. Galaczi

Centre for Research in English Language Learning & Assessment (CRELLA)
,
Cambridge English Language Assessment

Research Output:

Chapter in Book/Report/Conference proceeding

Chapter

Peer-review

Open access

Abstract

The rapid advances in speech processing and machine learning technologies have attracted language testers’ strong interest in developing automated speaking assessment in which candidate responses are scored by computer algorithms rather than trained human examiners. Despite its increasing popularity, automatic evaluation of spoken language is still shrouded in mystery and technical jargon, often resembling an opaque "black box" that transforms candidate speech to scores in a matter of minutes. Our chapter explicitly problematizes this lack of transparency around test score interpretation and use and asks the following questions: What do automatically derived scores actually mean? What are the speaking constructs underlying them? What are some common problems encountered in automated assessment of speaking? And how can test users evaluate the suitability of automated speaking assessment for their proposed test uses? In addressing these questions, the purpose of our chapter is to explore the benefits, problems, and caveats associated with automated speaking assessment touching on key theoretical discussions on construct representation and score interpretation as well as practical issues such as the infrastructure necessary for capturing high quality audio and the difficulties associated with acquiring training data. We hope to promote assessment literacy by providing the necessary guidance for users to critically engage with automated speaking assessment, pose the right questions to test developers, and ultimately make informed decisions regarding the fitness for purpose of automated assessment solutions for their specific learning and assessment contexts.

Publication Information

Output type

Research Output:

Chapter in Book/Report/Conference proceeding

Chapter

Peer-review

Original language

English

Pages from-to (Number of pages)

Pages 333-343

Publication milestones

Published - 10/02/2021

Publication status

Published - 10/02/2021

Publisher

Springer, Japan, India, Australia, Germany, United States, United Arab Emirates, Austria, Switzerland, Italy, China, United Kingdom, Netherlands, Brazil, France, Singapore