Loading the benchmark…

Dataset

Google FLEURS test split, streamed via Hugging Face (not bulk-downloaded), CC-BY-4.0. 15 clips per language, kept in a 3–13s duration band, taken in stream order (not reshuffled) — so this is a convenience sample of the first matching clips, not a random draw from the full test set.

Sample size

90 clips total, 15 per language. Small enough that per-language confidence intervals are wide — they're shown next to every number below rather than hidden. Treat single-percentage-point differences as noise; treat the Nordic-language gap as a hard fact (it's a capability check, not a sample statistic).

Normalization

Lowercased, punctuation stripped, whitespace collapsed, applied identically to reference and hypothesis before computing word error rate. Diacritics are kept (they're phonemically meaningful in these languages). Numbers are not normalised: "25" vs "twenty-five" or "23:35" vs "23 35" counts as an error, which inflates WER for whichever model formats differently from the reference on number-heavy clips (e.g. en_010, it_012). FLEURS is clean read speech and may overlap with either model's training data — treat it as a smoke test, not a ranking.

WER

Corpus-level: total (substitutions + deletions + insertions) ÷ total reference words, computed with a standard word-level edit-distance alignment.

Confidence intervals

95% interval from 2,000 clip-level bootstrap resamples per language.

Models

@cf/deepgram/nova-3 and @cf/openai/whisper-large-v3-turbo, both via Cloudflare Workers AI — not Deepgram's or OpenAI's own hosted APIs directly. Numbers reflect this specific hosting path, not necessarily either vendor's best-available accuracy.

Not included

Deepgram Flux — it's a turn-taking, WebSocket-only conversational model built for live voice agents, not a batch transcription endpoint, so it doesn't fit a WER comparison like this one. A voice-agent latency/turn-taking demo would be the right way to evaluate it, and is a natural next step.

Live "try it" box

Whisper large-v3-turbo is genuinely live. Nova-3's live entry is disabled: its Workers AI binding (used from inside the Function) rejects every binary audio shape tried (ArrayBuffer, Uint8Array, Blob, byte array) with the same schema error the raw REST API gives for a JSON body — while the identical audio sent as a raw HTTP POST with a Content-Type header works fine, which is how the precomputed benchmark itself was run. Fixing the live box would mean calling that REST endpoint from the Function with a Cloudflare API token as a server-side secret, which isn't wired up in this deployment. Rate-limited (10/hour per visitor, ~300/day globally) via the Cache API — best-effort and per-datacenter, not a hard global guarantee. Audio duration isn't verified server-side; a 5 MB upload cap stands in for a duration check. Separately: Cloudflare's own free-tier Workers AI account has a 10,000-neuron/day cap, which this project's benchmark run alone came close to — a live demo running on a free account can hit Cloudflare's limit before it hits either of this app's own rate limits.

This is the shape of a real proof-of-concept, just with public data standing in for a customer's recordings:

1. Get 30–60 minutes of the customer's own audio across the conditions that matter to them — accents, background noise, call quality, domain vocabulary — not a public benchmark. Public sets like FLEURS are clean read speech; real support calls, meetings or IVR audio behave differently.

2. Agree on ground truth before running anything: either the customer's existing transcripts, or a small hand-corrected reference set, with the normalization rules written down up front (case, punctuation, numbers, filler words) so nobody re-litigates the scoring after seeing the numbers.

3. Report WER per segment that matters to the business (per queue, per agent, per language) rather than one blended number, plus latency and cost at the volume they'll actually run — a model that wins on accuracy but not on their concurrency needs isn't a win.

4. Say what's uncertain. A 90-clip public benchmark like this one is a starting point for a conversation, not a procurement decision — and I'd rather tell a customer their sample is too small to be sure than round a confidence interval away.