Skip to content
AIOS Scribe logoAIOS Scribe
Article

What 278 Automated Transcript Reviews Found, and What They Cannot Tell You

6 min readReviewed QualityDataAI transcription

Every transcript AIOS Scribe stores is reviewed by a set of automated checks. We started keeping the results on 15 October 2026, and as of 19 October there were 278 of them. This post reports what they found, in plain numbers, and is just as clear about what the numbers cannot tell you. If you use AI transcripts for anything you will quote or publish, the pattern of what gets flagged is more useful than any claim about overall accuracy.

What the sample is, and is not

The 278 reviews cover about 270 different videos that were run through the service since it began transcribing inside the app. A large share were chosen by us to test it: songs, long talks, videos in other languages, a few that we knew had caused trouble. They are not a random sample of YouTube, and they are not a measure of how an average visitor's video will do. We did not listen to the audio of these videos to check them, which is why none of the numbers below is an accuracy figure.

The headline: most transcripts are flagged, a few are held

Of the 278 reviews, 274 finished. Of those:

  • 88 (32%) raised no flags at all.
  • 186 (68%) were flagged for review: something looked worth a human's attention.
  • 3 failed to run and 1 was deferred, so those are not counted as passes.

A flag is not an error. It means a rule noticed a pattern that is sometimes a mistake and sometimes perfectly normal. A transcript is held back from new requests only for a short list of serious problems, such as broken or interleaved passages. On 17 October four transcripts were in that state.

What gets flagged most

CheckWhat it looks forResult across the reviews
RepetitionThe same run of words repeated close togetherFlagged in 131 of 278 (47%)
CoverageA long stretch with no speech, or speech that stops earlyFlagged in 87 of 278 (31%)
TimestampsTimes in order and inside the videoFailed in 55 of 278 (20%)
Speech speedImplausibly dense speech in a 15-second windowFlagged in 13 of the 104 where it could be measured
Coherence (an AI reviewer)Broken, interleaved or repeated clausesWarned in 15, failed in 5, of 274
Accuracy against a referenceWord accuracy needs an audio-verified transcriptNot assessed in any of the 278

The last row matters most. Nothing in this review establishes that the words are right, because we have no human-corrected reference to compare against. What we have and have not measured spells that out.

Reading each flag sensibly

Repetition: often a song, sometimes a loop

Repetition fired in nearly half the reviews, and 91 transcripts had five or more repeated windows. In a song or a chant that is usually the chorus. In a talk it can be a speaker restating a point. It can also be a model looping, which is the case worth catching. Text alone cannot tell these apart, which is why we flag the repeat and do not delete it: we tried removing repeats automatically and took it out again, because it deleted genuine choruses.

Coverage: silence is a legitimate reason

The median transcript covered essentially the whole video, and 222 of 274 covered at least 90% of it. But 85 transcripts had a single quiet stretch longer than 30 seconds, and 60 longer than a minute. A long intro, music or a pause explains many of those; a model that skipped a section explains some. The check cannot say which, so it asks a person to look.

Timestamps: mostly cosmetic, not all

A fifth failed the timestamp check, which sounds alarming. When we examined a set of 270 on 17 October, the failures were mostly cosmetic: zero-length segments, an end a few seconds past the length of the video, and one real bug we had already fixed, where the AI switched from seconds to minutes-and-seconds partway through (1:07 arriving as 1.07). Because that last one is repaired when a transcript is read, we do not hold a transcript for timestamps alone; we flag it.

Speech speed: a sanity check, not a score

Across the transcripts the median was 141 words a minute, with 10% below 47 (music-heavy or sparse videos) and 10% above 246 (fast or dense speech). A window above 320 words a minute is flagged as implausible. Several scripts, such as Chinese, cannot be measured this way, which is why the check was assessed on only 104.

What to do with this as a reader

  1. Treat an unflagged transcript as “no automated concerns”, never as “correct”.
  2. Treat a flag as a pointer to a specific place to check against the audio, using the timestamp.
  3. Check any quote before you publish it, flagged or not. That advice does not change with the review result.
  4. Be most careful with music, several languages in one video, and very fast or overlapping speech.

You can see one real result, including a flag and what it means, on the worked example. The checks and their limits are described in how we evaluate transcription quality, and every change to them is dated on the what's new page.

What we are still missing

We have not yet compared any transcript with a human-corrected one, so we cannot say how often the words are wrong. That benchmark is the next real piece of work, and when it exists we will publish the result here, including if it is unflattering.

Need a transcript to try this on?

Paste a public YouTube link and get a full transcription with timestamps. Free within fair-use limits, with a free account.

Try AIOS Scribe