Developing speaking test tasks that elicit the construct of English language proficiency for employability

Abstract

English being the global language of business, the corporate sector worldwide is keen to find more effective ways to assess the English language proficiency (ELP) of potential recruits. Testing ELP for employment purposes is therefore becoming increasingly important both for businesses and for test developers. Available language tests have been criticised as too expensive, too generic, or otherwise unfit-for-purpose (Lockwood, 2012a; Lockwood and Raquel, 2019). This study is specifically concerned with the business processing outsourcing (BPO) industry in Oman and investigates:(a)The speech functions (SFs) used by customer service representatives (CSRs) in the TLU domain of a call centre,(b)Whether the language used in the speaking test designed for this study or the job interview in current use better connects with that used in the TLU domain. Following a sequential mixed method research approach, a research design with three distinct stages is followed. Stage 1 analyses the TLU domain, identifying SFs used by CSRs. Sources of data include a collection of 73 authentic calls (i.e., 2663 interactional turns) between CSRs and clients, interviews with stakeholders, document analysis, and observation. Following Bachman and Palmer's (2010) test development framework, Stage 2 involves the development of a speaking test based on the findings from Stage 1. Stage 3 involves audio recordings of performances by 29 CSR candidates in two speech events (speaking tests and job interviews). The SFs elicited through each speech event are compared using an observation checklist (OC) adapted from O'Sullivan, Weir, and Saville (2002) consisting of 28 SFs. This is followed up by the administration of two questionnaires eliciting CSR candidates' and examiners' views of the test. Qualitative data are analysed using thematic analysis and the quantitative data are analysed using descriptive statistics and functional analysis at two levels followed by a statistical test at each level. Wilcoxon's Signed Rank Test is used to test the statistical significance of the difference in the overall number of SFs elicited by the two speech events. This indicates that the number of SFs employed by CSR candidates in the speaking test is greater (Md = 20) than the number of SFs employed by the same CSR candidates in the interview (Md = 12), Z = -4.603, and that this difference is statistically significant (p < .0001).McNemar's Chi Square Test is used to test the statistical significance of the difference in the frequency of elicitation of individual SFs. McNemar's test results show that the test elicits more instances of SF use than the job interview for 12 SFs while the job interview elicits more instances of SF use than the test for 4 SFs (p

Date of Award	1 Mar 2023
Original language	English
Awarding Institution	University of Bedfordshire
Supervisor	Nahal Khabbazbashi (Supervisor) & Anthony Green (Second supervisor)