What an AI Actually Scores in a Spoken English Test

What Does an AI English Test Actually Score?

Two vendors that publish how their scoring works both say it scores text rather than sound. The recording is turned into words by automatic speech-to-text, a language model rates those words, and the rating is set against scores human experts gave to answers on similar questions. One vendor publishes that three-stage sequence and states its AI relies only on what is said ¹.

That statement goes further and names what is left out: no video analysis, no other audio characteristics, and specifically no facial expressions, body language, background or tone of voice ¹.

A second vendor's candidate help page says its AI reviews only the transcript on audio and video questions ².

Both are company pages about their own scorers, not audits, and both describe scoring answers in general rather than a dedicated language test.

Both statements are also narrower than they sound. Neither says the recording goes unwatched: on the second platform the audio and video are transcribed and handed to the employer along with the recording, so a person can still play it ². And relying only on what is said is a claim about the scoring model rather than about accuracy, because the words themselves arrive through a speech recognizer.

The label on a sub-score is a separate problem. A reporter at MIT Technology Review took one vendor's phone interview twice. Answering normally in English returned 8.5 out of 9 for English competency. Answering entirely in German, by reading a Wikipedia entry aloud, returned 6 out of 9, and a repeat run in German returned the same score ³.

Two-thirds of an English score went to speech containing no English.

That is one reporter, one vendor, in 2021, and the vendor's reply was that the system had not been tested in German before. Read it as a demonstration that a label can mean less than a candidate assumes, not as an error rate for the industry.

Where Does the Error Come From?

The transcription step is where the measured error sits. A rating built on text inherits every word the recognizer got wrong, and measured error is not spread evenly across speakers. Across four commercial speech-to-text services, mean word error rate was 52.6% for deaf and hard-of-hearing speakers against 5.0% for hearing speakers, about ten times higher ⁴. Those rates come from 484 recordings by 24 deaf or hard-of-hearing speakers and nine hearing ones ⁴.

The average hides the finding underneath it. In the same paper, error on speech rated highly intelligible sat close to the hearing group, while error on speech rated low sat far above it ⁴.

That was laboratory reading, and no published work turns a transcription error rate into a competency score.

Disordered speech shows a similar shape. Among 432 people with self-reported disordered speech, a commercial speech-to-text service reached a median word error rate of 31.5% on short recorded phrases ⁵. Those were short set phrases rather than spontaneous interview answers. Models personalized to one speaker did far better in the same study, but personalization needs hundreds of recordings of that person's own voice, so a tool meeting you for the first time cannot use one. Those models date from 2021 and current systems do better.

Accent is a narrower question. Across three large cloud recognizers, information lost from the transcript was 0.14 lower for speakers whose first language is English, measured on 2,713 people reading one identical paragraph ⁶.

That metric is not a word error rate and does not convert into one.

A non-native accent is not automatically a wall, though. On 2,400 single-sentence read recordings from 24 speakers whose first languages were Arabic, Chinese, Hindi, Korean, Spanish and Vietnamese, the best systems reached mean match error rates of 0.054 and 0.056, which the authors describe as approaching human accuracy ⁷. Six well-resourced first languages is not every accent, match error rate is not word error rate, and that paper is a preprint.

Read speech from a pronunciation corpus is close to the easiest condition there is.

A recorded interview answer is close to the hardest, on whatever microphone and room you have. Whether an AI interview tool can score you down for your accent or a disability follows the same pipeline one step further, into the scoring model.

Ask What Level of English This Job Requires

Ask the recruiter what the score is used for and what the job needs spoken English to do. The EEOC said in 2016 guidance that an English fluency or proficiency requirement is generally permissible only where it is required for the effective performance of the position it is imposed on ⁸.

The same guidance says applying one fluency standard across a broad range of dissimilar positions, or requiring a greater degree of fluency than a position needs, may violate Title VII ⁸. It illustrates the point with a person proficient enough to work as a research assistant but not yet fluent enough for a senior scientific writing role. That is agency guidance rather than a statute, it applies in the United States, and it was written about stated requirements rather than about a machine transcript.

On accent the standard is narrower still. A decision may legitimately rest on an accent only where the accent interferes materially with job performance, and the employer has to show both that effective spoken English is required for the duties and that this person's accent materially interferes ⁹.

A discernible accent and one that interferes are not the same thing.

No source here settles whether that standard reaches a machine's misrecognition.

There is a quieter cost worth naming. In a two-week diary study, 93% of 30 African American participants, all native English speakers, reported changing how they spoke to be understood by voice technology ¹⁰. That study covers consumer devices rather than hiring, and a group that size describes itself rather than a population. But performing a different voice for a scoring model is work, and it is not what the test claims to reward.

Hiring teams meet a version of this question from their own chair, in how to judge communication after an AI-translated interview, where a candidate's words arrive through a machine. The answer there is to name the requirement in the posting and test it openly.

Set Up the Accommodation Before You Record

Set it up before you record, because the remedy is a change to how the test is given rather than a re-read afterwards. Under the ADA's implementing regulation, an employer must administer employment tests so that, for an applicant whose disability impairs sensory, manual or speaking skills, the results reflect the skill the test is meant to measure rather than the impairment ¹¹.

The exception sits inside the rule, where those skills are what the test measures.

So a role that genuinely requires spoken English is a harder case than one where speech is only the input channel. That regulation is live but narrow. The separate duties to provide a reasonable accommodation and not to screen out qualified applicants sit elsewhere in the ADA and its regulation.

On at least one major platform the route is a button shown before the interview. HireVue's statement reproduces a sample Request Accommodations prompt, which employers can reword, that relaxes time limits for a qualified disability, and says the request goes to the relevant department of the employer, which then decides whether an exemption is granted ¹². On that platform the relaxed limits also remove the game-based section, so ask what changes as well as what is relaxed.

That platform also treats a submitted answer as final. Its candidate help center says you cannot return to a question once a response is submitted, and only the hiring company can authorize a new attempt ¹³.

Name the barrier and tie it to a disability or a medical reason in the same sentence. A request that never says why the change is needed reads as a preference, and no diagnosis is needed to make the reason clear.

• Say: "the recorded speaking test is a barrier for me because of a disability; is a written or live alternative available"
• Ask which speech-to-text engine produces the transcript, and whether a person reviews a low result
• Ask what the score is used for at this stage, and whether it works as a cutoff

This has a counterpart written for the people who field the request. Whether ADA accommodations apply to an AI-based assessment, and what an employer offers is what hiring teams read on the same question, and how to ask for an accommodation on an AI interview covers the timing.

Keep a copy of what you asked for and the date you sent it.

See what gets scored >