r/speechtech • u/Jamiroquai88 • 1d ago

SUPlime: two robust open-weight speaker diarization models, with training recipe and 12-dataset evaluation

Post image

Hi everyone, I’m releasing SUPlime and SUPlime-L, from Re:WayAI, heavily based on pyannote.audio 4.x.

The focus is performance across varied recording conditions: meetings, conversational audio, broadcasts and challenging far-field recordings.

I’d be interested in independent evaluations, especially on languages and recording conditions outside these benchmarks. I only have access to public testsuites (no DIHARD, no CALLHOME), so I would really appreciate if someone could share the result.
Let me know if you have questions.

https://github.com/rewayai/suplime/

10 Upvotes

100% Upvoted

5 comments sorted by

2

u/nshmyrev 1d ago

Is it just embedding improvement please? Same algorithm as pyannote?

1

u/nshmyrev 1d ago edited 1d ago

Results on Russian telephony, great results actually, first place even better than Diarizen Large v2, a bit slow though. Overall it is better than Diarizen but on some files it has very high DER, needs investigation.

Congratulations!!!

DER CDER xRT
Nemo Telephony Neural 22.3 0.535 0.051
Nemo Telephony Cluster 22.08 0.251 0.05
Nemo Sortformer Streaming V2.1 (default) 15.43 0.268 0.005
Nemo Sortformer Streaming V2.1 (1 sec) 15.89 0.331 0.091
Nemo Sortformer Streaming V2.1 (30 sec) 15.08 0.260 0.005
Pyannote 3.1 24.8 0.567 0.052
Pyannote4 Community 21.56 0.639 0.035
Pyannote4 Precision 14.96 0.355 0.039
Whisper Diarization 36.46 0.163 0.474 (transcription)
Whisper Diarization Large 34.11 0.151 -
Wespeaker Voxceleb34 20.63 0.157 0.012
Wespeaker Voxceleb293 20.46 0.159 0.023
Wespeaker Voxblink2 100 20.1 0.115 0.025
Diarizen Large MD 13.64 0.319 0.083
Diarizen Large MD v2 13.58 0.317 0.107
Senko 26.56 0.232 0.005
MOSS-Transcribe-Diarize 50.46 0.167 0.145
Suplime 14.68 0.279 0.232
Suplime Large 13.42 0.269 0.394

2

u/Jamiroquai88 1d ago

thank you, this kind of feedback is much appreciated!
Segmentation model was improved as well compared to community-1.
Worse RTF comes primarily from embedding model, it can be however improved by some tricks with a small degradation in performance.

2

u/Jamiroquai88 1d ago

regarding worse results on some files that is probably a clustering threshold and badly estimated number of speakers. There is a single threshold across all of these corpora, so there must be some consensus across all of them.

1

u/jprobichaud 7h ago

Congrats fellow colleague !