How to Score a TTS Bake-Off When the Demo Is Lying to You

The demo always sounds perfect.

That's the tell.

A vendor walks you through a thirty-second clip of a warm, unhurried voice reading a script someone hand-tuned the night before. It sounds human. Of course it does. Then you put the same model behind a live banking line, at 4 p.m. on a Tuesday, reading an account number a caller just rattled off in a mix of Arabic and English, and the wheels come off in a way the demo could never show you.

If your RFP still opens with a MOS score, you are grading the one thing that no longer separates anyone.

The metric you can stop caring about

Here is the uncomfortable truth about naturalness in 2026: it's basically a solved problem, and the leaderboards prove it. On the Artificial Analysis Speech Arena, which ranks models by Elo from blind human A/B votes, the top of the table is a traffic jam. Qwen-Audio-3.0-TTS-Plus at 1236, Simba 3.2 at 1234, Gemini 3.1 Flash TTS at 1214, Cartesia Sonic 3.5 at 1207. That's the field's best models packed inside roughly 30 Elo points. An 82-million-parameter open model, Kokoro-82M, topped a naturalness arena over rivals five to fourteen times its size. You can now hit the naturalness ceiling with a model that runs offline on a laptop.

And MOS itself — the mean-opinion-score panel that still anchors most procurement decks — is worse than uninformative when systems get this good. Academic work through 2025 keeps making the same point: MOS has limited resolution as models approach human quality, high inter-rater variance, and scores that swing depending on whether raters hear isolated sentences or full paragraphs. Human-parity claims "often fail under deception testing." You are asking a panel to split hairs on a dimension where the hairs are already indistinguishable.

So stop leading with it. Naturalness is table stakes now. Grade the things that still vary by an order of magnitude.

What actually varies 10x

Three things, mostly: how fast the first audio arrives under real load, whether the model reads your strings correctly, and whether it holds up in a live back-and-forth. None of these show up in a demo, because a demo is a single request, on a clean script, from a co-located machine, with no one talking back.

Start with latency, because it's the one everyone thinks they've measured and almost no one has.

Advertised latency is a marketing number. It's best-case, single-request, same data center. Measured latency is what your caller actually feels, and the gap is embarrassing. When Coval ran an independent probe of production endpoints on May 4, 2026 (surfaced via Gradium), ElevenLabs Flash v2.5 clocked a P50 time-to-first-audio of 288ms against a marketed figure near 135ms. Cartesia Sonic-3 came in best of the major streamers at 188ms. And OpenAI's batch TTS-1-HD landed at 2,295ms — over two seconds before the first sound, which is a warning about which endpoint you actually wired up as much as about the vendor.

Then there's the number that hides the killer.

The mean latency tells you how the call sounds in the brochure. The P95 and the jitter tell you how it sounds at 4 p.m.

Averages lie by smoothing over the tail. A model with a decent mean and a 380ms interquartile range — which is what Coval measured on Rime Mist-v3 — will still produce calls that feel broken, because it's the occasional laggy turn that makes a caller start talking over the bot. Report P50 and P95, and report the spread. A tight, boring 250ms beats a flashy 150ms mean with a fat tail every single time.

Building the test set from your own traffic

This is the part teams skip, and it's the part that decides everything.

The vendor benchmarks its model on clean text. Your production text is 1/4, $5.7M, PRM423GDDML2354, a drug name no phonetic dictionary has ever seen, and a date written the way your CRM happens to store it. Text normalization is the most common production failure mode in TTS, and it's completely invisible until it isn't. 1/4 with no context could be "January the fourth," "April first," "one fourth," or "one slash four." A model that wins the naturalness arena can still say your customer's balance wrong, confidently, in a beautiful voice.

So don't build your test set from benchmark boilerplate. Build it from a corpus of your real utterances:

• Pull a few hundred actual turns from your support and sales transcripts — the messy ones, not the tidy ones.
• Over-sample the entities that carry risk: account numbers, currency amounts, IBANs, dates, reference codes, product SKUs, medication and policy names.
• Include the code-switching you actually see. If half your callers move between Gulf Arabic and English mid-sentence, your test set has to.
• Add the noise. Mobile mics, background chatter, the acoustic reality of a call, not a studio.

If your benchmark dataset doesn't sound a little embarrassing, it isn't representative. Real customers sound messy, and the whole point is to fail the model on the same inputs your customers will.

The measurements that belong in the protocol

Once you have the corpus, the scoring is mechanical. A few checks do most of the work, and each one has a threshold you set before you listen, so you're grading against a bar instead of talking yourself into whichever voice you liked best.

Round-trip intelligibility. The cheapest automatic proxy for "did it say the entity correctly" is to run the synthesized audio back through a good speech-to-text engine and diff it against the source text. TTS in, ASR out, compare. It won't catch every prosody sin, but it catches the ones that matter for compliance: a mangled account number or a dropped decimal shows up as a word error immediately. On Coval's run, word error rates clustered low — Flash v2.5 at 5.2%, Aura-2 at 6.4% — but those are on generic text. Run it on your entities and the spread widens fast.

Entity and normalization accuracy on your strings. Score this separately and score it hard, because it's the one that erases trust on the first high-stakes transaction. Some vendors have built for exactly this: Rime's Mist line is deterministic and IVR-focused, with a spell() control for reading codes letter by letter and best-in-class number and currency handling. Deepgram claims Aura-2 hits 89% "good" accuracy on enterprise edge cases against rivals' 53–75% — though that's Deepgram's own marketing, so treat it as a hypothesis to test, not a fact to quote. Either way, the number that counts is the one you measure on your account numbers, not theirs.

Barge-in and turn-taking. A voice agent lives or dies on the interruption. The 2026 production bar is now quantified: barge-in detection under ~400ms from speech onset, false-barge-in rate under 2%, missed true interruptions under 1%, TTS flush under 60ms, and a total round-trip of roughly 800ms for the conversation to feel natural. Note that this is a pipeline number, not a TTS number. A 90ms TTS sitting behind a sluggish LLM is pointless. You're benchmarking the whole stack — speech-to-text, reasoning, synthesis, and the barge-in flush together — which is exactly why the serious money is moving to Coval-style simulation that replays thousands of noisy callers on every model swap, borrowed straight from how autonomous-vehicle teams test.

Determinism. For regulated voice, expressiveness is often a liability. In banking or telco you need the same balance, the same drug name, the same confirmation code pronounced identically on every call, because auditability demands reproducibility. This is where the flagship expressive models quietly disqualify themselves. ElevenLabs shipped v3 to general availability in March 2026 with bracketed audio tags and a claimed 68% cut in complex-text errors — and explicitly excluded it from real-time use, because "there is no way to get v3 quality at Flash speeds." A model that improvises emotion is the wrong tool where a compliance officer needs last month's call to sound like this month's.

The line item most scorecards forget

The same three-second voice clone that impresses in a demo is an active attack surface the moment you deploy voice at scale.

The FBI logged more than 22,000 reports of AI voice and video scams in 2025, with reported losses near $893M. One CFO authorized a $243,000 transfer after a call that perfectly mimicked the CEO — a cloned deepfake. "Can it clone a voice from a short clip" should read as a red flag in your threat model, not a feature bullet. Which is why watermarking now belongs in the scorecard: every output from Gemini 3.1 Flash TTS ships watermarked with SynthID by default. Put consent controls, liveness, and provenance on the same page as latency, or you'll grade a model that sounds wonderful and opens a hole.

Keep the humans, add the machines

None of this replaces a listening panel — it reframes it.

Automation tells you whether you kept quality after the last deploy. A round-trip WER check, a distribution-based objective metric like TTSDS2 (the only one of sixteen tested metrics to clear a Spearman of 0.50 across every domain, which makes it a credible alternative to paying for MOS panels), a latency probe running in CI — these catch regressions the moment a vendor ships a new endpoint. Native-speaking testers tell you something automation can't: whether a Gulf Arabic speaker accepts this voice as legitimate for a bank, the "almost right" failures that tank trust and never show up in a spectrogram. We go deep on why that judgment is so hard to automate in our piece on dialects and the voice-quality gap; this article is its companion — the how-to-measure-it.

Humans anchor what "good" means in a culture. Machines anchor whether you held onto it. You need both, and you need them on a loop, because agentic systems change often and a one-time pick rots.

Here's the shape of a scorecard that grades what matters:

TTS bake-off scorecard (weight to your risk, not the vendor's demo)

Naturalness (blind pairwise, your listeners) ....... pass/fail gate, not a tiebreaker
P50 time-to-first-audio, your region, under load ... target and measured
P95 time-to-first-audio + jitter (IQR) ............. the tail is the number that matters
Round-trip intelligibility (TTS -> STT -> diff) .... WER on YOUR entities, not generic text
Entity/normalization accuracy ...................... % correct on account #s, $, dates, drug names
Barge-in detection / flush / round-trip ............ <400ms / <60ms / ~800ms, end-to-end
Determinism (same string, same output) ............. required for regulated voice
Voice-security posture ............................. watermarking, consent, liveness, cloning risk

Weight the rows to your reality. An IVR that reads balances all day weights determinism and entity accuracy over expressiveness. A sales assistant weights latency and warmth. There is no universal winner, which is the whole reason the provider landscape is a menu and not a ranking.

The only benchmark that counts

The best-sounding model in the world can still be the wrong pick, and the leaderboard will never tell you that, because the leaderboard is grading naturalness — the one thing everyone already nailed.

A useful benchmark is built from your prompts, your noise profile, your entities, and your definition of acceptable. It measures the first audio under load, reports the tail and not the mean, checks that the model reads your account numbers the same way twice, and treats voice cloning as a risk instead of a bragging right. That's less exciting than a slide with a single big number on it. It's also the difference between a voice that impresses your steering committee and one that survives a Tuesday afternoon in production. If you want to argue thresholds over the specifics — and tie them back to KPIs that actually mean something — that's the conversation we'd rather be having anyway.

Bring your worst transcripts. That's where the real benchmark starts.

Updated on 15 July 2026.