We Benchmarked Whisper Tiny in the Browser: 4.3% WER, 13x Realtime, and One Surprise

Every captioning tool claims to be accurate and fast. Almost none of them publish a number, a method, or anything you could check. So we measured our own pipeline and are publishing the results, including the parts that do not flatter us.

MakeCaption runs OpenAI's Whisper `tiny.en` model directly in your browser. This is the smallest English-only Whisper variant — the trade-off that makes a ~74 MB download and on-device processing viable at all. Here is what that trade-off actually buys and costs.

Method

We generated six clips from a fixed 27-word script using three system voices (en_GB, en_US and en_AU) at two speech rates (150 and 195 words per minute). Each was converted to 16 kHz mono PCM — the exact format our pipeline feeds the model — then transcribed with `chunk_length_s: 30` and `stride_length_s: 5`, with word-level timestamps requested.

• Model: `Xenova/whisper-tiny.en` via Transformers.js
• Hardware: Apple M1, single machine, CPU inference
• Clips: 6 (3 voices x 2 speech rates), 7.0-8.9 seconds each
• Accuracy metric: word error rate (WER) — Levenshtein distance over words against the known script, after lowercasing and stripping punctuation
• Speed metric: realtime factor — audio duration divided by processing time

Results

• Mean word error rate: 4.3% — roughly one wrong word in every 23
• Mean speed: 13.1x realtime — a 60-second clip transcribes in about 4.6 seconds
• Zero-duration word timings: 2.5% of all words (4 of 159)
• Out-of-order word boundaries: 0% — no word started before the previous one ended

Per-voice, the spread is the interesting part. The en_GB voice scored 0.0% WER at both speech rates — a perfect transcript twice. The en_US voice scored 3.7% at both rates. The en_AU voice scored 7.4% and 11.1%, getting worse as speech got faster.

What the accent gap means for you

This is a known property of Whisper rather than a defect in our implementation: the training data skews toward North American and British English, and accuracy degrades on accents further from that centre. The `tiny` model has the least capacity to absorb that variation, so it shows the effect most starkly.

The practical implication is simple. If your speech sits close to the training distribution, expect to skim the transcript and fix the occasional proper noun. If it does not, budget real editing time — and do not trust any tool, ours included, that tells you automatic captions are a finished product.

The 2.5% that justifies a whole extra processing stage

The zero-duration figure is the one that drove a design decision. Whisper predicts timestamps as part of its output sequence rather than measuring them from the waveform, so a word can come back with a start and end that are identical — a caption that is on screen for exactly zero seconds.

At 2.5% of words, that is roughly one broken timing every 40 words. For sentence-level subtitles it is invisible. For the word-by-word karaoke highlighting that short-form video relies on, it is immediately obvious — a word that never lights up, or one that flashes and vanishes.

This is why our pipeline runs a refinement pass after transcription: we compute a short-time energy envelope over the extracted audio and snap suspect word boundaries to nearby onsets. Where no confident onset exists we leave the model's estimate alone rather than inventing precision. We explain that stage in more detail in how browser-based video captioning works.

Limitations — read these before quoting the numbers

We would rather publish caveats than have you over-trust a small experiment:

• These are synthesised voices, not human speech. System TTS is cleaner than real recordings — no room tone, no overlapping speakers, no microphone variation. Expect real-world WER to be worse, not better.
• Six clips is a small sample. The per-voice figures are directional, not statistically robust.
• One machine. Speed on an Apple M1 says nothing about a mid-range Android phone, where the same work may run several times slower.
• Clean, common vocabulary. The script contains no technical jargon, product names or unusual proper nouns — the exact categories where ASR fails hardest.
• `tiny.en` is the floor, not the ceiling. Larger Whisper variants score substantially better; they are simply too large to download into a browser tab for a one-off job.

Why publish this at all

Because "AI-powered" and "highly accurate" are not claims, they are decoration. A 4.3% word error rate on clean audio, an 11-point accent gap, and one broken timestamp every 40 words are claims — checkable, reproducible, and occasionally unflattering.

If you want to reproduce this, everything you need is above: the model id, the sampling rate, the chunk and stride settings, and the metric definitions. You can also just run a video through the tool and read the transcript yourself — including the sample clip on the homepage, which needs no upload.