Why Whisper Cuts off Indic Transcripts after Six Seconds

When we set the same 30-second VAD window for English and for the Indic languages, we observed something strange: the Indic transcripts would stop abruptly, mid-sentence, and the last visible character was almost always a weird question mark — the replacement character � (U+FFFD), the Unicode fallback for a byte sequence that cannot be decoded.

We did a detailed root cause analysis, documented both the cause and the fix, and shipped it into our pipeline. This article is the public version of that writeup.

Is it a training-data problem

When we first hit this, our instinct was that it came from the data: the audio clips available for training and fine-tuning Indic models are generally very short, so perhaps the models had simply learned to stop early. We later found that GenSRT, a public subtitling project, had stumbled onto the exact same wall and reached the exact same first guess.

That guess turns out to be wrong. GenSRT's own investigation ran a parameter sweep and got byte-identical output across thirteen different configurations. The truncation is deterministic, and it comes from a much deeper place and GenSRT measured it hitting roughly 8% of their chunks. (As an aside, GenSRT now ships our Indic models in their registry; our released datasets and models live in the Vividh-ASR collection.)

Root cause: the output token cap

Whisper is a transformer that takes in exactly 30 seconds of audio — anything shorter is padded out with zeros — and produces a sequence of tokens, up to 448 of them. Those 448 positions are shared by the special tokens that mark the start of a transcript, the timestamp tokens, and the actual text tokens.

How those 448 positions are split is where the different implementations diverge. OpenAI's reference decoder reserves half of them for previous text, so that each window can be conditioned on the prediction history. You can see this directly in decoding.py, where the sample length is capped at n_text_ctx // 2, i.e. 224 newly generated tokens, leaving the other 224 for the prompt. Yoad Snapir's long-form write-up is a good walkthrough of this prompting mechanism, and the limit itself is discussed here. faster-whisper, the CTranslate2 wrapper, inherits the same static halving — see this CTranslate2 issue. Hugging Face Transformers behaves differently: it caps prompt + new tokens at 448 (positional limit, prompt budget), so in the absence of any prior text it can emit up to 448 new tokens.

So the effective cap on freshly generated tokens is 224 for OpenAI and faster-whisper/CT2, and 448 for Hugging Face Transformers. The 448 ceiling itself is fixed by the architecture; it cannot be raised.

Why Indic hits the limit and not English

A token cap of 224 sounds generous — for English. The reason it bites Indic languages so early is tokenization.

Whisper reuses GPT-2's byte-level BPE tokenizer directly, as the Sony Research India work points out. In English, the tokenizer has learned plenty of merges, so a word is usually one, two, or at most three tokens — each token is a subword. But the Indic scripts are almost absent from the data that tokenizer was trained on, so very few byte-level merges exist for them. With no merges to fall back on, every character is emitted as its raw UTF-8 bytes — and every Indic scipt character is three bytes, hence three tokens. The same roughly three-tokens-per-character figure shows up in the bit-level BPE and fertility benchmark studies, and BharatGen notes it is most severe for exactly these scripts.

In other words: for English, one token is a subword; for Hindi or the other Indic languages, one token is a sub-character. A 224-token budget that holds a long English sentence holds only a fraction of the equivalent Hindi.

If the 224-token cap happens to land in the middle of a character's three-byte sequence — say after the first two of three bytes — the transcript ends on an incomplete, invalid UTF-8 fragment. There is no character there to render, so the decoder substitutes the � replacement symbol. That is the weird question mark we kept seeing at the end of our Indic transcripts. The same mid-grapheme failure is described here and this blogpost captures the essence with an associated playground.

The quick fix — shorter windows

The most direct fix is to never reach the cap: split the audio into chunks short enough that the transcript stays under 224 tokens. From our measurements, a voice-activity detection (VAD) window of six or seven seconds keeps Indic transcripts safely below the limit, so the abrupt � never appears. It works with stock faster-whisper by cutting the audio into pieces far shorter than the thirty seconds Whisper was designed to hear at once.

The deeper fix — a higher cap

The better fix is to stop giving away half the budget in the first place. The 224 limit is not the architectural ceiling — Whisper can decode 448 tokens; OpenAI's implementation simply reserves the other half for a previous-text prompt that our segmented pipeline never uses. Removing that static halving is a one-line change in our CTranslate2 fork, and it lets the decoder emit up to 448 new tokens, so we can keep the longer windows without corrupting the output.

At five seconds the transcripts almost never reach 224 tokens, so lifting the cap barely improves the error rate. At ten seconds, the transcript tokens routinely overflowed in the baseline making the error rate worse than at five. But with the fix in place, the extra headroom improves the error rate by 7 points partly from the truncation it prevents, and partly from the longer window's added context, which Whisper turns into a more accurate guess. The numbers come from our SCRIBE ASR evaluation framework on the Vividh-ASR benchmark dataset — 26 hours of complexity stratified Malayalam test subset.

The real fix

Neither fix touches the root cause. More context makes Whisper transcription measurably better, and more still would help — but many of the world's languages can rarely reach for it while running on a tokenizer built primarily for English. The imbalance is not in the algorithm — a byte-level BPE is formally language-agnostic — but in the data it learns from: an English-heavy corpus yields an English-heavy vocabulary however neutral the method.

As pointed out in the broken-token article, only about 0.00165% of GPT-3's training data was Malayalam; the digitized text that exists for Indic languages is measured in gigabytes where English is measured in terabytes. The three-tokens-per-character penalty, and the truncated transcripts it produces, are the far end of that shortage.

A better tokenizer removes one compounding factor, but the durable fix is upstream of all of them — more digital content being produced and made available in the world's diverse languages, so that the tokenizer has something better to learn from. And where the data is slow to accumulate, the algorithm can be made to meet these languages halfway — tokenizers that are aware of a scripts characteristics rather than blind to it, so that a scarce corpus is spent more wisely as in the recent exploration by NVIDIA and Gnani AI. Both are slow roads. Until then, that small � at the end of a cut-off transcript stands in as an indicator of how much of the world's writing is still penalized by tools built for English.