Whisper, GPT-4o, Gemini, Grok or Azure: which transcribes best?

A few months ago I had to give a presentation at work, and I hate hearing my own voice. I went looking for a tool that would replace my audio, drop the filler words, and keep the timing exactly, so the mouse on screen still matched the narration. I did not find one that fit, so I had an AI write me a rough script that did, and the presentation went out in a voice that was not mine.

A few weeks later I turned that script into Recastr, partly to see how far I could get building a side project with Claude Code. The first thing that broke was transcription accuracy. Before committing to a provider I ran a benchmark, and this post is what it found. The harness, the fixtures and the raw results are in parkerroan/transcription-benchmark on GitHub, and there is a longer chapter-by-chapter writeup of the pipeline if you want to see how each number was produced. A shorter version of this post ran on Medium in August.

What was compared

Eight models from five vendors, in two families.

Dedicated speech-to-text models, built to transcribe:

• whisper-1, gpt-4o-transcribe, gpt-4o-mini-transcribe and gpt-4o-transcribe-diarize from OpenAI
• grok-stt from xAI
• azure-fast-transcription from Microsoft

General multimodal chat models, given the audio and a "transcribe this" prompt:

• gpt-audio from OpenAI
• gemini-3.7-flash from Google

Every model id was checked against the vendor's live model list on 2026-08-13 rather than assumed. Two I had planned to include, gpt-4o-audio-preview and gemini-2.5-flash, had already been retired by then.

I needed two things from a transcript that a word error rate cannot tell you about: a timestamp on every word, so each subtitle cue can start where the speaker started it, and speaker labels, so a two-person recording can get two voices. Only three of the eight return word timestamps at all, and only one of those three also labels speakers natively.

Model	Vendor	Word timestamps	Native diarization	Streaming	Steerability	Measured $/min
whisper-1	OpenAI	Yes	No	No (batch only)	Vocabulary hint	$0.0060
gpt-4o-transcribe	OpenAI	No	No	Yes	Vocabulary hint	$0.0035
gpt-4o-mini-transcribe	OpenAI	No	No	Yes	Vocabulary hint	$0.0018
gpt-4o-transcribe-diarize	OpenAI	No (segment level)	Yes	Yes	Vocabulary hint	$0.0192
gpt-audio	OpenAI	No (promptable, unreliable)	Promptable	Text tokens only	Full prompt	$0.0214
gemini-3.7-flash	Google	No	Promptable	Separate Live API	Full prompt	$0.0021
grok-stt	xAI	Yes	Yes	Yes	Structured params	$0.0017
azure-fast-transcription	Microsoft	Yes	Yes (up to 35 speakers)	Separate real-time tier	Structured params	$0.0060

Cost per minute is what the benchmark measured, not the list price. The two chat models are billed per token, so their rate moves with how much they say.

The clips

Eight fixtures, in two deliberately different groups.

The six synthetic clips were generated with OpenAI's tts-1, so the ground truth is exact by construction: clean narration, a clip dense with numbers and proper nouns, technical jargon, narration with white noise laid over it, and scripted two- and three-speaker dialogues. They answer the easy question, whether a model can transcribe at all.

The two real recordings answer the harder one. A two-minute excerpt from the Library of Congress's 1949 interview with Fountain Hughes, conducted by Hermond Norwood, and a five-minute excerpt from NASA's "Houston We Have a Podcast" episode 414, a modern studio interview with an official transcript. They are 77 years apart on purpose. If only the 1949 clip were hard, this would be an "old audio" story. It is not: every model degrades on both.

Both clips are public domain, so here they are as the models heard them.

0:000:00

0:000:00

A third real clip, a 1962 White House Dictabelt recording, was tried and dropped. One speaker was barely audible even to a person, which makes it a poor source of ground truth whatever a model does with it.

How each transcript is scored

Every clip goes to all eight models, and the whole pass repeats ten times before anything is averaged, because model output and API latency both vary call to call. That is 80 calls per model and 640 in all. Each transcript is aligned to the reference words in text order with jiwer, which gives the word error rate. The numbers-and-names and jargon clips also report a key-term miss rate, the share of the important words that came back wrong. Latency is wall-clock time per call.

Diarization needs one extra step. Provider speaker labels are arbitrary, xAI returns integers and the prompted models return strings like "Speaker 1", and none of them line up with the reference's A and B. The scorer tries every mapping between the two label sets and reports the best one, so a model is never penalised for calling the first speaker "2".

The results

Every number below is a mean over the ten runs. The table is sorted by synthetic word error rate, best first.

Model	Synthetic WER	1949 interview	2026 podcast	Latency	$/min
gpt-audio	5.1%	29.8%	10.7%	3.4 s	$0.0214
azure-fast-transcription	7.1%	33.1%	13.1%	3.3 s	$0.0060
gemini-3.7-flash	7.2%	33.9%	20.3%	11.6 s	$0.0021
gpt-4o-transcribe	7.2%	33.7%	22.0%	3.2 s	$0.0035
gpt-4o-mini-transcribe	7.9%	33.0%	11.3%	1.9 s	$0.0018
whisper-1	8.5%	30.9%	12.2%	4.0 s	$0.0060
grok-stt	8.9%	35.6%	23.3%	0.9 s	$0.0017
gpt-4o-transcribe-diarize	15.4%	33.9%	15.7%	22.1 s	$0.0192

Three things stand out.

gpt-audio, a chat model, is the most accurate transcriber in the set. It has the lowest error rate on every group of clips, 5.1% synthetic, 29.8% on the 1949 interview and 10.7% on the podcast. It wins outright, not on average. It also costs about six times what gpt-4o-transcribe costs per minute.

Real audio is two to four times harder than synthetic audio for every model, and the modern podcast is not exempt. gpt-4o-transcribe-diarize degrades the least, about 1.6x, and gpt-audio and gpt-4o-transcribe the most, about 4x. The 1949 interview is the hardest clip for all eight, usually by a wide margin.

gpt-4o-transcribe-diarize sits at the bottom of the synthetic column because of one clip. It scored 53.8% on the numbers-and-names fixture, the worst single result in the benchmark, while managing 3.5% on the two-speaker dialogue. If your audio is full of figures, that matters.

Cheap and fast is not the same as cheap and accurate. grok-stt is the cheapest model at $0.0017 a minute and the fastest by a factor of two, with a mean latency of 0.9 seconds. It also has the worst error rate of any model on both real recordings, 35.6% on the interview and 23.3% on the podcast. gpt-4o-mini-transcribe costs almost the same, $0.0018 a minute, and halves that podcast error rate to 11.3%. On accuracy per dollar it is the clear pick, as long as you do not need timestamps or speakers.

Diarization

Five of the eight models label speakers. Three never emit a label at all, whatever the audio.

Model	Synthetic dialogues	1949 interview	2026 podcast
azure-fast-transcription	96.1%	92.3%	92.3%
gemini-3.7-flash	100%	85.6%	91.9%
gpt-4o-transcribe-diarize	99.8%	85.2%	84.6%
gpt-audio	100%	83.3%	88.5%
grok-stt	84.0%	73.7%	92.3%

azure-fast-transcription is the most consistent diarizer, at 92.3% on both real clips with no swing between eras. grok-stt swings the widest, from 73.7% on the 1949 interview to 92.3% on the podcast, a 19-point gap. The two chat models are perfect on the clean scripted dialogues and drop into the mid-80s on the interview.

One result changed under repetition. A single run had put gpt-4o-transcribe-diarize among the worst on real audio. Averaged over ten, it lands mid-pack and is fairly consistent, about 85% on both clips. That is the argument for running a benchmark more than once before believing it.

Picking a model depends on what you need

There is no single best model here. Each row below is called strictly from the numbers above.

If you care most about	Pick	Why
Cost	grok-stt	$0.0017 a minute, cheapest by a clear margin
Accuracy, ignoring speakers	gpt-audio	Lowest WER on every clip group
A timed SRT with speakers	azure-fast-transcription	Word timestamps and native diarization, and the flattest diarization across eras
Noisy audio	gemini-3.7-flash	Tied at 0% WER on the white-noise clip with four others, and the cheapest of the five
Speakers on clean audio	gpt-audio or gemini-3.7-flash	Both 100% on the scripted dialogues
Names, numbers and jargon	gemini-3.7-flash or gpt-4o-transcribe	Lowest key-term miss rate, 10.8%
Latency	grok-stt	0.9 s mean, twice as fast as the next model and about 24x faster than gpt-4o-transcribe-diarize
Accuracy per dollar	gpt-4o-mini-transcribe	$0.0018 a minute with 11.3% on the podcast, against grok-stt's 23.3%

What Recastr uses

Recastr turns a recording into a voiceover that lands on the original cues, so it needs a timestamp per word and a label per speaker. That narrows the field to one model, and it is also the steadiest diarizer in the set, so azure-fast-transcription is the primary. grok-stt and whisper-1 sit behind it as fallbacks, in that order, because they are the other two models that return word timestamps. If you want to see what that transcript becomes, the screen recording voiceover page walks through the transcribe, edit and re-voice steps, or you can drop a file into the studio.

Caveats

These numbers are a snapshot from 2026-08-13. Models get updated, prices change, and none of these models is deterministic, so a re-run will not reproduce them exactly. Ten runs smooth the noise; they do not remove it. gemini-3.7-flash failed two of its 80 calls outright and those calls are excluded from its means. The cost column is what the API charged for these clips, which for the token-billed chat models depends on the audio as much as on the price list.