6.4%.
That number once thrilled me. On a Japanese semiconductor-technology video, Google Chirp 3 produced text fairly close to the subtitles I used for comparison. The other configuration recorded a much larger difference on the same material.
I called it a leap “from usable to commercially ready.” Looking back, several necessary checks were missing between an attractive score and that conclusion.
My need was concrete: when Japanese customers discuss delivery dates, quality problems, and next steps, the tool should aid understanding rather than lead me astray with fluent-looking text. That need remains. But a test must first explain what it measured.
Is the reference transcript actually correct?
I selected four Japanese videos, took the first three minutes of each, sent them through two recognition configurations, and compared their output with YouTube’s automatically generated subtitles. I called the resulting metric CER, or character error rate.
This calculation typically compares recognized text against a reference, counts substitutions, deletions, and insertions, and divides by the number of reference characters. Crucially, the formula does not judge whether my reference transcript is correct.
Automatic subtitles can mishear too. Two systems may make the same mistake; a recognizer may also be right but lose points for disagreeing with an incorrect subtitle. Normalization of Japanese kanji, kana, numbers, and punctuation also affects the comparison.
Until someone checks the reference against the audio, I would rather describe these values as character-level differences from the reference subtitles. They describe how two texts differ; they cannot directly establish how many characters out of every hundred were misheard.
What the original results can still tell me
The table preserves the values recorded in the original article. This is not a newly executed test. Each column represents a complete configuration, including its segmentation and prompting settings.
| Material | Groq Whisper configuration | Chirp 3 configuration |
|---|---|---|
| International-news commentary | 52.5% | 9.2% |
| Semiconductor business analysis | 61.6% | 25.5% |
| Semiconductor technology commentary | 36.5% | 6.4% |
| Business conversation | 40.6% | 13.0% |
In these four comparisons, the Chirp 3 configuration consistently produced text closer to the reference subtitles. That observation is worth preserving and gives me a reason to test it further.
It does not mean four kinds of situations were adequately represented. Each category contained only one clip. Online explanatory videos may not represent real meetings with overlapping speakers, unstable recording quality, or dense terminology. This revision did not retrieve the audio, a human-verified reference, and complete scoring records, or rerun the comparison. Reproducibility of the recorded values remains to be checked.
Averaging the four values and calculating an improvement percentage does not fill those gaps. Nor can a relative reduction in an error metric simply be renamed an equal percentage improvement in accuracy.
I compared two configurations
The original Groq configuration used five-second segments; the Chirp 3 configuration used fifteen-second segments and approximately a thousand business terms. The model, amount of speech context, and prompting changed together.
The table cannot separately establish how much came from the model, the terminology list, or longer segments. To isolate one factor, other conditions would need to be held fixed while comparing its presence and absence.
Comparing complete configurations still matters to users. I am choosing a record that ultimately works, not a model name detached from the workflow. With the conditions explained, we can discuss whether a combination deserves further adoption testing.
Google’s current Chirp 3 documentation lists language, method, and feature support. Implementation still requires checking the selected region, method, and features. My historical configuration numbers should not be treated as universal specifications today.
Which words matter in a business record?
I built the “Agora” real-time meeting-recording tool to improve cross-language work through AI collaboration. But when expressions such as 取引先, 発注書, and ご指導のほど were misrecognized, the text interfered with understanding. Those original experiences explain my motivation better than a winner’s chart.
For acceptance, I would examine both completeness and the words that change action: company and personal names, delivery dates, quantities, amounts, negation, and conditions.
A transcript can get most characters right yet turn “not yet approved” into “approved.” An average score may look good while the work goes wrong. Conversely, penalties for punctuation or character forms may not have the same decision impact.
A more useful trial would use representative meeting material with permission and first create a human-verified reference. Fix the material and normalization rules, then compare complete configurations. Change segmentation or terminology separately when investigating causes. Finally, measure correction time instead of looking only at a percentage.
Count correction work alongside fees
The original article assumed US$0.016 per minute and calculated US$0.96 per hour. I retain that as a historical calculation assumption, not a current quote. Procurement requires checking the actual service and billing terms.
More importantly, I withdraw the promise that paying that amount makes every term correct or eliminates manual correction. Four clip comparisons did not establish either claim.
A useful cost comparison should put recognition fees, translation fees, waiting time, and correction work on the same work order. Longer segments may yield more complete text but impose too much delay for live captions. If a configuration reduces correction time, there is then a basis for estimating its value.
Free versus paid is only one condition. A paid product winning a small test does not make free tools a trap.
Choose for work risk, not a reassuring button label
I previously used routing by language and operating mode, with Japanese business mode selecting Chirp 3 and other situations using other engines. This describes the design at that time; I have not re-audited the current routing or providers’ plans.
The useful principle is matching configuration to work. Captions for quickly understanding the gist can have different latency and acceptance requirements from minutes prepared for customer confirmation. Neither should be considered verified simply because a button says “business mode.”
I am still willing to pay reasonably for cross-language communication. But the next time I see 6.4%, I want to open the reference transcript and listen again to the sentence that determines the delivery date.


💬 Comments
Loading...