SPEARBench Detailed Report

Summary paragraph

mini-omni is a strong performer on intelligibility and conversational smoothness: ASR-based CER/WER are much lower than the original speech on both improvised and naturalistic data, and latency plus interruption counts are also improved. Its main tradeoff is verbosity, since generated answers are substantially longer than the originals even when speech quality is better by UTMOS. Language behavior is very stable, with answers essentially entirely in English, and dialect behavior appears consistent across subsets, with similar entrainment and variance in improvised and naturalistic speech. On stance, the model largely preserves the original sign of the affective judgments, but it shifts slightly more positive than negative overall, indicating a mild warmening bias rather than a strong change in direction. The report mentions emotion and explainable-feature analyses, but does not provide concrete directional conclusions from those signals. Overall, mini-omni looks faithful in language and dialect, strong in intelligibility, and slightly biased toward warmer delivery while being longer and not clearly better at paralinguistic fidelity.

Data, Models, and Setup

Data

metric	Improvised dev	Improvised test	Naturalistic dev	Naturalistic test	Total
Unique original dialogues	270	181	480	511	1442
Selected dialogues	1165	994	1529	1731	5419
Hours of questions	4.78h	3.76h	6.76h	7.47h	22.77h
Hours of original answers	2.67h	2.29h	5.23h	4.36h	14.56h
Original total hours	7.45h	6.05h	12.00h	11.83h	37.33h
Hours of generated answers	3.56h	3.18h	4.86h	5.43h	17.03h
Mean original answer length	8.26s	8.28s	12.32s	9.07s
Mean generated answer length	11.04s	11.56s	11.51s	11.35s

Models

parameter	value
LLM model used for inference	mini-omni
ASR models used	Qwen3-ASR-0.6B, whisper-large-v3
Language ID model	facebook/mms-lid-126
Dialect ID model	tiantiaf/voxlect-english-dialect-whisper-large-v3
LLM used for stance	gpt-audio-1.5
Statistical tests used for p-values	Mann-Whitney U Test
UTMOS model	tarepan/SpeechMOS:v1.2.0 utmos22_strong
VAD model	silero-vad

Setup

parameter	value
protocol	seamless_2t_2s_questions
selection_method	end_with_question
min_turns	2
min_speakers	2
splits	dev,test
subsets	improvised,naturalistic
data_root	data/seamless_2t_2s_questions
results_root	results/seamless_2t_2s_questions

Intelligibility and Interruption Metrics

Metric	System	model_mean_naturalistic	original_mean_naturalistic	model_mean_improvised	original_mean_improvised
CER	Qwen3-ASR-0.6B	4.18	19.70	5.50	16.27
CER	whisper-large-v3	4.55	17.13	5.55	13.99
WER	Qwen3-ASR-0.6B	6.35	32.24	7.32	28.28
WER	whisper-large-v3	6.49	25.42	7.14	20.11
UTMOS		3.8641	2.1655	4.0254	2.3207
latency		112.4927	929.6360	118.2828	415.3924
average number of interruptions per dialogues		0.81	0.50	0.89	0.55
interrupted time (s)		1.166	0.520	1.378	0.522
number of dialogues with interruption		975/1721	545/1731	599/990	377/994

Intelligibility and interruption metric histograms with KDE

Basic metric histogram gallery

Turn Taking Surprisal

subset	metric	model_value	original_value	mean_diff	std_diff	p_value	n
improvised	mean_nll	2.105488173117549	3.392070671418477	-1.2866	1.6367	1.625e-51	509
improvised	tail_nll	2.105488173117549	3.520741825309412	-1.4153	1.7853	1.596e-51	509
improvised	dialog_nll	2.105488173117549	3.4564062483639444	-1.3509	1.7047	1.603e-51	509
improvised	naturalness_score	2.105488173117549	3.4564062483639444	-1.3509	1.7047	1.603e-51	509
naturalistic	mean_nll	2.08208827688539	3.9220984000715045	-1.8400	1.5511	8.329e-201	882
naturalistic	tail_nll	2.08208827688539	4.1670074320954695	-2.0849	1.7531	8.049e-201	882
naturalistic	dialog_nll	2.08208827688539	4.044552916083487	-1.9625	1.6409	8.082e-201	882
naturalistic	naturalness_score	2.08208827688539	4.044552916083487	-1.9625	1.6409	8.082e-201	882

Language and Dialect ID

subset	% Eng in models' answers	% Eng in original answers	second most spoken language in original answers	percentage 2nd language in original answers
improvised	100.0	99.73545	Vietnamese	0.088183
naturalistic	100.0	99.01332	dan	0.345338

Subset	Question/Answer	East Asia	English	Germanic	Irish	North America	Oceania	Other	Romance	Scottish	Semitic	Slavic	South African	Southeast Asia	South Asia
improvised	Question	0.404040	0.000000	0.101010	5.757576	82.020202	0.101010	0.000000	9.696970	0.101010	0.101010	0.202020	0.000000	0.505051	0.101010
improvised	Answer	2.854230	0.000000	0.305810	0.101937	85.626911	0.000000	0.000000	6.625892	0.000000	3.975535	0.000000	0.000000	0.509684	0.000000
naturalistic	Question	2.091807	0.522952	0.348635	4.474143	72.225450	0.232423	0.174317	16.618245	0.174317	0.348635	0.174317	0.348635	1.394538	0.348635
naturalistic	Answer	2.920561	0.000000	0.233645	0.233645	84.228972	0.000000	0.116822	4.147196	0.000000	7.476636	0.000000	0.000000	0.642523	0.000000

Dialectal Metrics

subset	metric	mean_diff	n	detail
improvised	Dialectal entrainment	0.4225	963	computed from dialect_logits.csv question_log_logits and answer_log_logits
improvised	Dialectal variance	138.8289	963	computed from dialect_logits.csv question_log_logits and answer_log_logits
naturalistic	Dialectal entrainment	0.4036	1669	computed from dialect_logits.csv question_log_logits and answer_log_logits
naturalistic	Dialectal variance	138.2398	1669	computed from dialect_logits.csv question_log_logits and answer_log_logits

Dialect question-to-answer confusion matrix

Dialect score spider profiles for question and answer fields

Emotional Naturalness

The relationship violin plot breaks naturalistic emotional naturalness logits down by relationship label.

Naturalistic emotional naturalness violins by relationship

The emotion scatter plot compares full-question and full-answer Arousal, Dominance, and Valence scores normalized to [-1, 1]. Dotted lines show per-dataset correlations with rho labels.

Question-answer Arousal, Dominance, and Valence scatter plots

STANCE

Stance Descriptions

stance	positive_original_%	negative_original_%	definition
Q0	81.9672131147541	18.0327868852459	Based on the audio, does the TARGET speaker sound warm and affiliative, or cold and detached?
Q1	46.31578947368421	53.68421052631579	Based on the audio, does the TARGET speaker sound compassionate and supportive, or callous and unsympathetic?
Q2	31.25	68.75	Based on the audio, does the TARGET speaker sound polite and respectful, or rude and disrespectful?
Q3	21.0	79.0	Based on the audio, does the TARGET speaker sound assertive and self-confident, or hesitant and inhibited?
Q4	57.142857142857146	42.857142857142854	Based on the audio, does the TARGET speaker sound straightforward and sincere, or sly and manipulative?
Q5	100.0	0.0	Based on the audio, does the TARGET speaker sound attentive and focused, or confused and distracted?
Q6	21.666666666666668	78.33333333333333	Based on the audio, does the TARGET speaker sound organized and goal-driven, or disorganized and unmotivated?
Q7	68.1159420289855	31.884057971014492	Based on the audio, does the TARGET speaker sound socially engaged and expressive with the other speaker, or withdrawn and disengaged?
Q8	50.0	50.0	Based on the audio, does the TARGET speaker accommodate and yield to the other speaker’s preferences, or do they try to control and dominate?
Q9	0.0	100.0	Based on the audio, does the TARGET speaker stay calm and avoid confrontation, or are they hostile and aggressive?