Spanish AI Intake Now Matches English Within 2.4 Points. Test the Other Two Things

On October 21 Sierra published τ-i18n, a benchmark built to answer one question about voice AI: does it still work when the caller is not speaking English? The team took its existing voice test — the same airline, retail and telecom tasks, the same policies, the same tools — and ran it in Spanish, Brazilian Portuguese, Hindi, Korean and Mandarin. Three agents, 2,700 calls, six languages, nothing else changed.

For anyone waiting for a non-English phone line to be good enough, the completion numbers are better than expected. Spanish trails English by 2.4 percentage points, Hindi by 3.8 and Brazilian Portuguese by 6.4, and the strongest agent finished 79.9% of tasks overall. On whether the job gets done, the gap is now too small to be the reason not to try.

The problem moved rather than disappeared

Sierra’s argument is that task completion hides the failures callers actually notice, so it scores two more things.

The first is interaction: does the agent take its turn at the right moment, stop talking when the caller starts, and ignore sounds not aimed at it. Two of the three agents missed a caller turn, answered slowly or failed to yield in more than 80% of calls. The third managed 16%. In between 26% and 60% of calls, agents replied more than three times to a backchannel — an “mm-hm”, a cough, somebody else in the room.

The second is generation quality, meaning what the caller actually hears, and it scores lowest of all: 63.4% for the best agent and 17.7% for the worst. It penalises translationese, grammatical-gender errors, the wrong register and missing honorifics — a female-voiced agent calling itself by the masculine form in Hindi, or a stiff “execute a cancellation operation” in Mandarin where a person would say “let me help you cancel”. An agent, as Sierra puts it, can complete the task while still falling short on the interaction or the language itself.

One detail that costs a firm the retainer

The sharpest example in the study is a form field. Korean and Mandarin callers usually give their family name first, and an agent that drops that into the given-name field can fail authentication outright — the caller said their name correctly and the system told them it did not match.

For a practice where the client’s language decides whether they phone at all — immigration, criminal defence, personal injury — that is not cosmetic. It arrives as a dead lead rather than a bug, which is why nobody finds it. Register failures are the quieter version: a prospective client handled in a tone that feels wrong hangs up and calls the next firm, and nothing in your intake report explains why.

What to test before you put an agent on a Spanish line

Make three calls to your own intake number in the language your clients actually use. One straightforward fee-and-practice-area question. One where you give a name that does not split cleanly into English first and last fields. One where you interrupt mid-sentence — on that 80% turn-taking figure, this is the test that fails.

Then check the plumbing: name order, accented characters, and whether a failed match ends in a message or a person. Write the escalation line in each language you answer, and route sensitive matters to an attorney.

Sierra open-sourced the language packs, and its release notes warn that 2.0 scores cannot be compared with the old ones — so treat any vendor number older than this week as unproven and make the calls yourself. The baseline to beat is not perfect Spanish. It is the voicemail a Spanish-speaking caller reaches at 9pm today, and an agent that answers every call and message in the caller’s language and books the consultation while they are still on the line clears that bar long before it clears any benchmark.