Why transcription tools struggle with overlapping speakers

What happens to a transcript when two people talk at once?

If you have ever run a lively meeting, panel, focus group, or family interview through a transcription tool and got back a transcript with missing words, merged sentences, or the wrong names attached to quotes, you have met the overlapping-speech problem. Usually the tool is not broken, and the audio is not unusable — the problem is genuinely hard for current systems.

Two systems are involved. Automatic speech recognition (ASR) converts audio into words. Speaker diarization answers the separate question of who spoke when, splitting the recording into segments and labelling each one with a speaker. Both systems are built around a quiet assumption: that one person is talking at a time. When two people talk over each other, that assumption breaks, and the transcript shows it in three ways — words are dropped, two voices are merged into a single line, or the words are credited to the wrong person.

If you are choosing a tool and care about speaker labels, see our guide to why speaker identification matters and our roundup of transcription tools with speaker identification. If you already have a transcript and need to check quotes that fall in busy passages, see how to verify quotes in an AI transcript.

Why does speech recognition miss overlapping words?

Most speech recognisers decode one dominant voice at a time. When a second person starts talking over the first, the recogniser tends to lock onto the louder or clearer voice and treat the quieter one as background noise — so the quieter speaker's words simply disappear from the transcript. In milder crosstalk, the recogniser may blend fragments of both voices into a single garbled sentence.

The effect is large and well documented. On the LibriCSS meeting test set (Chen et al., 2020), utterance-wise word error rate for unprocessed audio rose steadily as the overlap ratio increased:

In other words, going from no overlap to 40% overlap nearly quadrupled the error rate for the unprocessed condition, while separation into multiple channels before recognition kept errors far lower (8.4% to 21.7%). These figures come from 2020 Kaldi-era systems, so treat the trend, not the absolute levels, as the point: overlap degrades recognition badly, and separating the audio before recognising it recovers much of the loss.

This is not a new observation. Shriberg, Stolcke, and Baron (2001) found that recognition performance "degrades considerably" for overlapped speech in multi-party conversation — two decades before modern neural recognisers, and the pattern still holds.

Why do speaker labels go wrong during crosstalk?

Speaker diarization has the same single-voice assumption baked in. Many diarization systems cluster audio segments and assign each segment one speaker label, so a moment where two people talk at once has to be given to one of them — or to neither. The industry-standard way to measure this is diarization error rate (DER). Following the DIHARD III definition, DER is the sum of three components: missed speech (a speaker talked but no label was produced), false alarm (a label was produced but nobody talked), and speaker confusion (speech attributed to the wrong speaker). Overlapped speech is scored, with no forgiveness collar, so getting overlap wrong is penalised directly.

On meeting data, missed speech is typically the largest share of the error. The pyannote speaker-diarization-3.1 model card illustrates the pattern: on the AMI headset-mic condition it reports a DER of 18.8% (3.6% false alarm, 9.5% missed speech, 5.7% confusion), and on AMI far-field audio 22.4% (3.8% / 11.2% / 7.5%). On AliMeeting it reports 24.4%, on DIHARD 3 21.7%, and on VoxConverse 11.3%. In the meeting conditions, missed speech — the component that grows when people talk over each other — is the single biggest contributor.

That is why crosstalk so often shows up in a transcript as a dropped line rather than a mislabelled one: the diarizer simply declines to assign a speaker to audio it cannot split, and the recogniser has already dropped or merged the words anyway.

How much overlap is there in real conversations?

More than you might think. In the DIHARD III challenge data, roughly 8.75% of speech in the evaluation core and about 10% in the development core is overlapped — speech where more than one person is talking at the same time. Shriberg et al. (2001) found overlap to be common in both meetings and phone calls, not an occasional accident.

So if a quarter of your transcript's errors cluster in a tenth of its audio, that is not a coincidence — it is the overlap. Panel discussions, focus groups, brainstorming sessions, and family interviews are the worst cases, because turn-taking is fast, people interrupt enthusiastically, and there is often no chairperson enforcing order. The CHiME-7 DASR challenge (Cornell et al., 2023) built an entire evaluation around far-field, multi-device meeting transcription precisely because dinner-party-style conversation with frequent overlap remains one of the hardest settings in the field.

Do more speakers make transcripts worse?

Yes — both in published benchmarks and in device tests. Lanzendörfer et al. (2025) benchmarked diarization models across many scenarios and found that while the best system achieved a DER of 11.2% overall, failures were worse at high speaker counts, with missed speech the most common error type.

The same pattern shows up in English transcription tests from Shenzhen meeting rooms in August 2026. Each condition was a single recorded session, so compare only within a row:

From the 2-speaker interview (C02) to the 8-speaker meeting (C05), every device's error rate rose — between about 3 times (DingTalk A1, 18.5% to 56.2%) and 5.5 times (Plaud NotePin S, 9.6% to 53.2%). Notta Memo Pro followed the same pattern, placing second to Plaud Note Pro in every multi-speaker condition while staying well ahead of the Anker and DingTalk recorders at most speaker counts.

One caution: this data does not measure overlap, so do not read it as "overlap multiplies errors by five". The scenarios also differ in room and content — C02 was recorded in a small meeting room, C05 in an open training area — and the 1-speaker speech condition (C01) actually scored worse than the 2-speaker interview. The safe conclusion is that error rates rose with speaker count in these tests, consistent with the published research on crowded conversations. You can see the full device-by-device comparison in our Notta Memo vs Plaud Note Pro face-off and our test methodology.

How can I get a better transcript of a crowded conversation?

You cannot make people stop interrupting each other, but you can change what the tool has to work with. In rough order of impact:

• Record each person on a separate microphone or track. This is the single biggest fix, and it mirrors what the LibriCSS separation results show: splitting the audio before recognition recovered most of the overlap penalty. Most meeting tools cannot do it — in competitor research (checked September 2026), studio-quality multitrack recording is unavailable in most AI meeting tools. Descript is the standout exception, offering local/remote multitrack recording and a speaker-detection setting that can detect 8+ speakers. MacWhisper Pro takes a related approach for podcasts, with a beta feature for combining single-track audio recorded separately for each host.
• Record mic and system audio as separate channels where your tool allows. The Notta Memo Pro, for example, records its microphone and system audio as separate channels, which keeps your voice and a remote participant's voice on distinct tracks — helpful for hybrid calls, though it does not isolate each person in the room.
• Avoid speakerphone. A room picked up through a phone speaker compresses and muddles voices before the recogniser ever sees them.
• Seat people close to the recorder. Distance hurts the quiet voices first, and quiet voices are the ones that get dropped in overlap.
• Set the speaker count where the tool offers it. Some tools, including Descript, ask how many speakers to expect; giving the right number helps the diarizer.
• Ask people not to talk over each other on key points. You do not need a silent room — just ask that decisions, quotes, and numbers be stated one at a time.
• Human-check the busy passages. Jump to the overlap-heavy moments in the transcript and verify them against the audio. A tool like Notta, which syncs the transcript to the recorded audio so you can tap a line and hear the moment it came from, makes this check far quicker than scrubbing through a raw recording. For high-stakes recordings, human transcription remains the strongest fallback for crosstalk: Rev charges $1.99 per minute and HappyScribe offers human proofreading from $2.00 per minute.

If you are recording meetings or lectures regularly, our guides on how to record a Zoom meeting, phone vs app vs recorder for lectures, and the best AI voice recorders cover pickup quality in more depth. For interviews specifically, see our roundup of the best interview transcription software.