SPEARBench Detailed Report - Qwen2.5-Omni-7B

Summary paragraph

Qwen2.5-Omni-7B is strong on intelligibility, with lower CER/WER than the original answers on both improvised and naturalistic subsets, no recorded interruptions, and high UTMOS indicating clear speech. The main tradeoff is brevity and speed: generated answers are much shorter than the originals, and latency is very large on naturalistic data. Language ID shows the model stays overwhelmingly English, with only tiny spillover into other languages, while dialect behavior suggests moderate entrainment but substantial dialectal variance across questions and answers. Emotion/stance behavior is mostly stable, as the model preserves the original stance sign in most cases, though it shifts slightly more often toward positive or negative in a small fraction of examples. The explainable-feature section is present and covers f0, duration, and voiced-ratio analyses, but the provided text does not include the actual feature deltas, so only that signal can be noted.

Data, Models, and Setup

Data

metric	Improvised dev	Improvised test	Naturalistic dev	Naturalistic test	Total
Unique original dialogues	270	181	480	511	1442
Selected dialogues	1165	994	1529	1731	5419
Hours of questions	4.78h	3.76h	6.76h	7.47h	22.77h
Hours of original answers	2.67h	2.29h	5.23h	4.36h	14.56h
Original total hours	7.45h	6.05h	12.00h	11.83h	37.33h
Hours of generated answers	2.76h	2.20h	3.50h	3.97h	12.42h
Mean original answer length	8.26s	8.28s	12.32s	9.07s
Mean generated answer length	7.57s	7.13s	7.51s	7.41s

Models

parameter	value
LLM model used for inference	Qwen2.5-Omni-7B
ASR models used	Qwen3-ASR-0.6B, whisper-large-v3
Language ID model	facebook/mms-lid-126
Dialect ID model	tiantiaf/voxlect-english-dialect-whisper-large-v3
LLM used for stance	gpt-audio-1.5
Statistical tests used for p-values	Mann-Whitney U Test
UTMOS model	tarepan/SpeechMOS:v1.2.0 utmos22_strong
VAD model	silero-vad

Setup

parameter	value
protocol	seamless_2t_2s_questions
selection_method	end_with_question
min_turns	2
min_speakers	2
splits	dev,test
subsets	improvised,naturalistic
data_root	data/seamless_2t_2s_questions
results_root	results/seamless_2t_2s_questions

Intelligibility and Interruption Metrics

Metric	System	model_mean_naturalistic	original_mean_naturalistic	model_mean_improvised	original_mean_improvised
CER	Qwen3-ASR-0.6B	6.83	19.70	5.78	16.27
CER	whisper-large-v3	9.77	17.13	6.43	13.99
WER	Qwen3-ASR-0.6B	21.43	32.24	20.65	28.28
WER	whisper-large-v3	17.85	25.42	17.16	20.11
UTMOS		4.1920	2.1655	4.1907	2.3207
latency		277.0984	929.6360	277.8179	415.3924
average number of interruptions per dialogues		0.00	0.50	0.00	0.55
interrupted time (s)		0.000	0.520	0.000	0.522
number of dialogues with interruption		0/1930	545/1731	0/1109	377/994

Intelligibility and interruption metric histograms with KDE

Basic metric histogram gallery

Turn Taking Surprisal

subset	metric	model_value	original_value	mean_diff	std_diff	p_value	n
improvised	mean_nll	4.950075606732063	3.392070671418477	1.5580	2.0325	3.687e-45	357
improvised	tail_nll	5.5229182920229976	3.520741825309412	2.0022	2.4079	2.057e-49	357
improvised	dialog_nll	5.23649694937753	3.4564062483639444	1.7801	2.1887	2.142e-48	357
improvised	naturalness_score	5.23649694937753	3.4564062483639444	1.7801	2.1887	2.142e-48	357
naturalistic	mean_nll	4.739199792538576	3.9220984000715045	0.8171	2.0310	8.837e-27	581
naturalistic	tail_nll	5.1748177971705855	4.1670074320954695	1.0078	2.4110	1.364e-24	581
naturalistic	dialog_nll	4.9570087948545805	4.044552916083487	0.9125	2.1920	5.279e-26	581
naturalistic	naturalness_score	4.9570087948545805	4.044552916083487	0.9125	2.1920	5.279e-26	581

Language and Dialect ID

subset	% Eng in models' answers	% Eng in original answers	second most spoken language in models' answers	percentage 2nd language in models' answers	second most spoken language in original answers	percentage 2nd language in original answers
improvised	99.461883	99.73545	Japanese	0.179372	Vietnamese	0.088183
naturalistic	98.581560	99.01332	jav	0.455927	dan	0.345338

Subset	Question/Answer	East Asia	English	Germanic	Irish	North America	Northern Irish	Oceania	Other	Romance	Scottish	Semitic	Slavic	South African	Southeast Asia	South Asia
improvised	Question	0.347222	0.086806	0.086806	5.555556	80.121528	0.000000	0.086806	0.000000	9.027778	0.086806	0.086806	0.347222	0.000000	0.347222	0.086806
improvised	Answer	6.041479	0.180343	0.721371	0.901713	65.464382	0.270514	0.000000	0.901713	10.640216	0.000000	0.000000	0.811542	0.000000	14.066727	0.000000
naturalistic	Question	1.945525	0.535019	0.291829	4.231518	68.968872	0.000000	0.243191	0.194553	15.807393	0.097276	0.243191	0.243191	0.340467	1.264591	0.243191
naturalistic	Answer	5.960946	0.000000	0.513875	0.822199	65.621788	0.411100	0.000000	0.565262	11.510791	0.000000	0.000000	0.565262	0.000000	14.028777	0.000000

Dialectal Metrics

subset	metric	mean_diff	n	detail
improvised	Dialectal entrainment	0.2955	937	computed from dialect_logits.csv question_log_logits and answer_log_logits
improvised	Dialectal variance	83.4265	937	computed from dialect_logits.csv question_log_logits and answer_log_logits
naturalistic	Dialectal entrainment	0.2825	1602	computed from dialect_logits.csv question_log_logits and answer_log_logits
naturalistic	Dialectal variance	85.0937	1602	computed from dialect_logits.csv question_log_logits and answer_log_logits

Dialect question-to-answer confusion matrix

Dialect score spider profiles for question and answer fields

Emotional Naturalness

The relationship violin plot breaks naturalistic emotional naturalness logits down by relationship label.

Naturalistic emotional naturalness violins by relationship

The emotion scatter plot compares full-question and full-answer Arousal, Dominance, and Valence scores normalized to [-1, 1]. Dotted lines show per-dataset correlations with rho labels.

Question-answer Arousal, Dominance, and Valence scatter plots

STANCE

Stance Descriptions

stance	positive_original_%	negative_original_%	definition
Q0	82.1917808219178	17.80821917808219	Based on the audio, does the TARGET speaker sound warm and affiliative, or cold and detached?
Q1	48.24561403508772	51.75438596491228	Based on the audio, does the TARGET speaker sound compassionate and supportive, or callous and unsympathetic?
Q2	32.142857142857146	67.85714285714286	Based on the audio, does the TARGET speaker sound polite and respectful, or rude and disrespectful?
Q3	21.428571428571427	78.57142857142857	Based on the audio, does the TARGET speaker sound assertive and self-confident, or hesitant and inhibited?
Q4	57.84313725490196	42.15686274509804	Based on the audio, does the TARGET speaker sound straightforward and sincere, or sly and manipulative?
Q5	100.0	0.0	Based on the audio, does the TARGET speaker sound attentive and focused, or confused and distracted?
Q6	23.076923076923077	76.92307692307692	Based on the audio, does the TARGET speaker sound organized and goal-driven, or disorganized and unmotivated?
Q7	68.35443037974683	31.645569620253166	Based on the audio, does the TARGET speaker sound socially engaged and expressive with the other speaker, or withdrawn and disengaged?
Q8	45.070422535211264	54.929577464788736	Based on the audio, does the TARGET speaker accommodate and yield to the other speaker’s preferences, or do they try to control and dominate?
Q9	0.0	100.0	Based on the audio, does the TARGET speaker stay calm and avoid confrontation, or are they hostile and aggressive?