Skip to search boxSkip to navigationSkip to main content

Opening the black box: exploring automated speaking evaluation

Research Output: Chapter in Book/Report/Conference proceeding Chapter Peer-review

Open access

Abstract

The rapid advances in speech processing and machine learning technologies have attracted language testers’ strong interest in developing automated speaking assessment in which candidate responses are scored by computer algorithms rather than trained human examiners. Despite its increasing popularity, automatic evaluation of spoken language is still shrouded in mystery and technical jargon, often resembling an opaque "black box" that transforms candidate speech to scores in a matter of minutes. Our chapter explicitly problematizes this lack of transparency around test score interpretation and use and asks the following questions: What do automatically derived scores actually mean? What are the speaking constructs underlying them? What are some common problems encountered in automated assessment of speaking? And how can test users evaluate the suitability of automated speaking assessment for their proposed test uses? In addressing these questions, the purpose of our chapter is to explore the benefits, problems, and caveats associated with automated speaking assessment touching on key theoretical discussions on construct representation and score interpretation as well as practical issues such as the infrastructure necessary for capturing high quality audio and the difficulties associated with acquiring training data. We hope to promote assessment literacy by providing the necessary guidance for users to critically engage with automated speaking assessment, pose the right questions to test developers, and ultimately make informed decisions regarding the fitness for purpose of automated assessment solutions for their specific learning and assessment contexts.

Publication Information

Output type

Research Output: Chapter in Book/Report/Conference proceeding Chapter Peer-review

Original language

English

Pages from-to (Number of pages)

Pages 333-343

Publication milestones

  • Published - 10/02/2021

Publication status

Published - 10/02/2021

Publisher

Springer, Japan, India, Australia, Germany, United States, United Arab Emirates, Austria, Switzerland, Italy, China, United Kingdom, Netherlands, Brazil, France, Singapore

ISBN (Print)

9789813342316

ISBN (Electronic)

9789813342323

Chapter Number

25

External Publication IDs

  • handle.net: 10547/624618
  • Scopus: 85151205019

Host publication title

Issues in Language Testing Around the World: Insights for Language Test Users.

Host publication editors

  • Betty Lanteigne
  • Christine Coombe
  • James Dean Brown

Access to documents

Publication metrics

Metrics

Download statistics
Download count
1