SPEARBench Detailed Report - Qwen3-Omni-30B-A3B-Instruct

Summary paragraph

Qwen3-Omni-30B-A3B-Instruct shows strong intelligibility, with much lower CER/WER than the original answers on both improvised and naturalistic subsets and high UTMOS, while also eliminating audible interruptions entirely in generated answers. The main tradeoff is latency: its responses are substantially slower than the originals, despite often being comparable or slightly shorter in length. Language behavior is highly consistent, with English dominating the generated speech and only small amounts of code-switching; dialectally, the model shows stable entrainment with high dialectal variance, suggesting some context adaptation but broad dispersion rather than a tightly controlled style. Emotion/stance behavior is mostly preserved, with the vast majority of examples matching sign and only a small tendency to shift more positive and slightly less negative than the originals. The source stance labels themselves are mixed across traits, so the results mainly indicate that the model tracks existing prosodic/affective cues rather than overriding them. Explainable speech-feature results are mentioned, but the report does not provide concrete feature differences, so no firm conclusion can be drawn about pitch, duration, or voiced-ratio behavior.

Data, Models, and Setup

Data

metric	Improvised dev	Improvised test	Naturalistic dev	Naturalistic test	Total
Unique original dialogues	270	181	480	511	1442
Selected dialogues	1165	994	1529	1731	5419
Hours of questions	4.78h	3.76h	6.76h	7.47h	22.77h
Hours of original answers	2.67h	2.29h	5.23h	4.36h	14.56h
Original total hours	7.45h	6.05h	12.00h	11.83h	37.33h
Hours of generated answers	3.70h	2.48h	4.22h	4.79h	15.20h
Mean original answer length	8.26s	8.28s	12.32s	9.07s
Mean generated answer length	10.17s	8.05s	9.07s	8.94s

Models

parameter	value
LLM model used for inference	Qwen3-Omni-30B-A3B-Instruct
ASR models used	Qwen3-ASR-0.6B, whisper-large-v3
Language ID model	facebook/mms-lid-126
Dialect ID model	tiantiaf/voxlect-english-dialect-whisper-large-v3
LLM used for stance	gpt-audio-1.5
Statistical tests used for p-values	Mann-Whitney U Test
UTMOS model	tarepan/SpeechMOS:v1.2.0 utmos22_strong
VAD model	silero-vad

Setup

parameter	value
protocol	seamless_2t_2s_questions
selection_method	end_with_question
min_turns	2
min_speakers	2
splits	dev,test
subsets	improvised,naturalistic
data_root	data/seamless_2t_2s_questions
results_root	results/seamless_2t_2s_questions

Intelligibility and Interruption Metrics

Metric	System	model_mean_naturalistic	original_mean_naturalistic	model_mean_improvised	original_mean_improvised
CER	Qwen3-ASR-0.6B	3.79	19.70	3.77	16.27
CER	whisper-large-v3	2.71	17.13	2.85	13.99
WER	Qwen3-ASR-0.6B	8.72	32.24	9.74	28.28
WER	whisper-large-v3	6.58	25.42	7.07	20.11
UTMOS		4.3246	2.1655	4.3609	2.3207
latency		26.2176	929.6360	29.0352	415.3924
average number of interruptions per dialogues		0.00	0.50	0.00	0.55
interrupted time (s)		0.000	0.520	0.000	0.522
number of dialogues with interruption		0/1930	545/1731	0/1109	377/994

Intelligibility and interruption metric histograms with KDE

Basic metric histogram gallery

Turn Taking Surprisal

subset	metric	model_value	original_value	mean_diff	std_diff	p_value	n
improvised	mean_nll	2.3406930301124595	3.392070671418477	-1.0514	1.7920	6.888e-23	331
improvised	tail_nll	2.343825159116094	3.520741825309412	-1.1769	1.9331	4.458e-23	331
improvised	dialog_nll	2.3422590946142767	3.4564062483639444	-1.1141	1.8565	5.110e-23	331
improvised	naturalness_score	2.3422590946142767	3.4564062483639444	-1.1141	1.8565	5.110e-23	331
naturalistic	mean_nll	2.3725711867759864	3.9220984000715045	-1.5495	1.7084	8.053e-97	627
naturalistic	tail_nll	2.3725711867759864	4.1670074320954695	-1.7944	1.8938	1.030e-98	627
naturalistic	dialog_nll	2.3725711867759864	4.044552916083487	-1.6720	1.7903	4.257e-98	627
naturalistic	naturalness_score	2.3725711867759864	4.044552916083487	-1.6720	1.7903	4.257e-98	627

Language and Dialect ID

subset	% Eng in models' answers	% Eng in original answers	second most spoken language in models' answers	percentage 2nd language in models' answers	second most spoken language in original answers	percentage 2nd language in original answers
improvised	98.513011	99.73545	Chinese	0.371747	Vietnamese	0.088183
naturalistic	97.926635	99.01332	Chinese	0.903775	dan	0.345338

Subset	Question/Answer	East Asia	English	Germanic	Irish	North America	Northern Irish	Oceania	Other	Romance	Scottish	Semitic	Slavic	South African	Southeast Asia	South Asia
improvised	Question	0.347222	0.086806	0.086806	5.381944	76.822917	0.000000	0.086806	0.000000	8.246528	0.086806	0.086806	0.347222	0.000000	0.347222	0.086806
improvised	Answer	1.226415	0.094340	0.188679	0.566038	89.811321	0.377358	0.188679	1.603774	3.962264	0.283019	0.188679	0.094340	0.000000	1.226415	0.188679
naturalistic	Question	1.702335	0.340467	0.243191	4.085603	65.223735	0.000000	0.243191	0.145914	15.175097	0.145914	0.291829	0.194553	0.291829	1.215953	0.291829
naturalistic	Answer	1.682953	0.000000	0.000000	0.977199	89.467970	0.217155	0.271444	1.194354	4.777416	0.162866	0.217155	0.000000	0.000000	0.868621	0.162866

Dialectal Metrics

subset	metric	mean_diff	n	detail
improvised	Dialectal entrainment	0.4871	902	computed from dialect_logits.csv question_log_logits and answer_log_logits
improvised	Dialectal variance	202.7209	902	computed from dialect_logits.csv question_log_logits and answer_log_logits
naturalistic	Dialectal entrainment	0.4806	1522	computed from dialect_logits.csv question_log_logits and answer_log_logits
naturalistic	Dialectal variance	205.0365	1522	computed from dialect_logits.csv question_log_logits and answer_log_logits

Dialect question-to-answer confusion matrix

Dialect score spider profiles for question and answer fields

Emotional Naturalness

The relationship violin plot breaks naturalistic emotional naturalness logits down by relationship label.

Naturalistic emotional naturalness violins by relationship

The emotion scatter plot compares full-question and full-answer Arousal, Dominance, and Valence scores normalized to [-1, 1]. Dotted lines show per-dataset correlations with rho labels.

Question-answer Arousal, Dominance, and Valence scatter plots

STANCE

Stance Descriptions

stance	positive_original_%	negative_original_%	definition
Q0	82.1917808219178	17.80821917808219	Based on the audio, does the TARGET speaker sound warm and affiliative, or cold and detached?
Q1	48.24561403508772	51.75438596491228	Based on the audio, does the TARGET speaker sound compassionate and supportive, or callous and unsympathetic?
Q2	32.142857142857146	67.85714285714286	Based on the audio, does the TARGET speaker sound polite and respectful, or rude and disrespectful?
Q3	21.428571428571427	78.57142857142857	Based on the audio, does the TARGET speaker sound assertive and self-confident, or hesitant and inhibited?
Q4	57.84313725490196	42.15686274509804	Based on the audio, does the TARGET speaker sound straightforward and sincere, or sly and manipulative?
Q5	100.0	0.0	Based on the audio, does the TARGET speaker sound attentive and focused, or confused and distracted?
Q6	23.076923076923077	76.92307692307692	Based on the audio, does the TARGET speaker sound organized and goal-driven, or disorganized and unmotivated?
Q7	68.35443037974683	31.645569620253166	Based on the audio, does the TARGET speaker sound socially engaged and expressive with the other speaker, or withdrawn and disengaged?
Q8	45.070422535211264	54.929577464788736	Based on the audio, does the TARGET speaker accommodate and yield to the other speaker’s preferences, or do they try to control and dominate?
Q9	0.0	100.0	Based on the audio, does the TARGET speaker stay calm and avoid confrontation, or are they hostile and aggressive?