SPEARBench Detailed Report

Summary paragraph

gpt-audio-1.5 performs very strongly on intelligibility, with much lower CER/WER than the original answers in both improvised and naturalistic speech, while also maintaining high audio quality by UTMOS. It is especially clean in turn-taking, recording essentially no interruptions compared with frequent interruptions in the originals, but this comes with a clear latency penalty because its responses are much slower. On language behavior, it stays almost perfectly aligned with the source language distribution, producing essentially all English and preserving the tiny amount of non-English content only in the originals. Dialectically, it shows moderate entrainment with substantial variance, suggesting some conversational adaptation without collapsing into a fixed dialect. For stance and emotion-related behavior, it is largely faithful to the originals with very high sign consistency, though there is a small tendency to shift more positive or more negative depending on the item rather than remaining perfectly neutral. The explainable-speech section is present, but the report text does not expose the underlying feature deltas, so no specific prosodic or temporal direction can be confirmed.

Data, Models, and Setup

Data

metric	Improvised dev	Improvised test	Naturalistic dev	Naturalistic test	Total
Unique original dialogues	270	181	480	511	1442
Selected dialogues	1165	994	1529	1731	5419
Hours of questions	4.78h	3.76h	6.76h	7.47h	22.77h
Hours of original answers	2.67h	2.29h	5.23h	4.36h	14.56h
Original total hours	7.45h	6.05h	12.00h	11.83h	37.33h
Hours of generated answers	3.59h	3.11h	4.70h	5.15h	16.55h
Mean original answer length	8.26s	8.28s	12.32s	9.07s
Mean generated answer length	9.86s	10.10s	10.10s	9.62s

Models

parameter	value
LLM model used for inference	gpt-audio-1.5
ASR models used	Qwen3-ASR-0.6B, whisper-large-v3
Language ID model	facebook/mms-lid-126
Dialect ID model	tiantiaf/voxlect-english-dialect-whisper-large-v3
LLM used for stance	gpt-audio-1.5
Statistical tests used for p-values	Mann-Whitney U Test
UTMOS model	tarepan/SpeechMOS:v1.2.0 utmos22_strong
VAD model	silero-vad

Setup

parameter	value
protocol	seamless_2t_2s_questions
selection_method	end_with_question
min_turns	2
min_speakers	2
splits	dev,test
subsets	improvised,naturalistic
data_root	data/seamless_2t_2s_questions
results_root	results/seamless_2t_2s_questions

Intelligibility and Interruption Metrics

Metric	System	model_mean_naturalistic	original_mean_naturalistic	model_mean_improvised	original_mean_improvised
CER	Qwen3-ASR-0.6B	1.45	19.70	1.69	16.27
CER	whisper-large-v3	1.21	17.13	1.29	13.99
WER	Qwen3-ASR-0.6B	6.03	32.24	6.57	28.28
WER	whisper-large-v3	5.22	25.42	5.61	20.11
UTMOS		4.3613	2.1655	4.3642	2.3207
latency		129.1343	929.6360	133.8753	415.3924
average number of interruptions per dialogues		0.00	0.50	0.00	0.55
interrupted time (s)		0.000	0.520	0.000	0.522
number of dialogues with interruption		0/1929	545/1731	0/1107	377/994

Intelligibility and interruption metric histograms with KDE

Basic metric histogram gallery

Turn Taking Surprisal

subset	metric	model_value	original_value	mean_diff	std_diff	p_value	n
improvised	mean_nll	2.965320577806134	3.392070671418477	-0.4268	1.7102	2.582e-05	509
improvised	tail_nll	2.9665725909491636	3.520741825309412	-0.5542	1.8539	1.135e-05	509
improvised	dialog_nll	2.9659465843776487	3.4564062483639444	-0.4905	1.7759	1.477e-05	509
improvised	naturalness_score	2.9659465843776487	3.4564062483639444	-0.4905	1.7759	1.477e-05	509
naturalistic	mean_nll	3.0084253058478843	3.9220984000715045	-0.9137	1.6281	6.288e-70	882
naturalistic	tail_nll	3.009416262419814	4.1670074320954695	-1.1576	1.8224	1.458e-72	882
naturalistic	dialog_nll	3.0089207841338492	4.044552916083487	-1.0356	1.7142	7.579e-72	882
naturalistic	naturalness_score	3.0089207841338492	4.044552916083487	-1.0356	1.7142	7.579e-72	882

Language and Dialect ID

subset	% Eng in models' answers	% Eng in original answers	second most spoken language in original answers	percentage 2nd language in original answers
improvised	100.0	99.73545	Vietnamese	0.088183
naturalistic	100.0	99.01332	dan	0.345338

Subset	Question/Answer	East Asia	English	Germanic	Irish	North America	Oceania	Other	Romance	Scottish	Semitic	Slavic	South African	Southeast Asia	South Asia
improvised	Question	0.347826	0.086957	0.000000	5.913043	83.391304	0.086957	0.000000	9.130435	0.086957	0.086957	0.347826	0.000000	0.434783	0.086957
improvised	Answer	0.000000	0.000000	0.000000	1.826087	97.043478	0.000000	0.000000	1.043478	0.000000	0.000000	0.000000	0.000000	0.086957	0.000000
naturalistic	Question	1.946472	0.583942	0.291971	4.622871	72.554745	0.243309	0.194647	16.934307	0.145985	0.291971	0.243309	0.340633	1.265207	0.340633
naturalistic	Answer	0.000000	0.000000	0.000000	1.265207	97.566910	0.048662	0.000000	0.973236	0.000000	0.000000	0.000000	0.000000	0.145985	0.000000

Dialectal Metrics

subset	metric	mean_diff	n	detail
improvised	Dialectal entrainment	0.4647	976	computed from dialect_logits.csv question_log_logits and answer_log_logits
improvised	Dialectal variance	97.4979	976	computed from dialect_logits.csv question_log_logits and answer_log_logits
naturalistic	Dialectal entrainment	0.4635	1688	computed from dialect_logits.csv question_log_logits and answer_log_logits
naturalistic	Dialectal variance	106.4159	1688	computed from dialect_logits.csv question_log_logits and answer_log_logits

Dialect question-to-answer confusion matrix

Dialect score spider profiles for question and answer fields

Emotional Naturalness

The relationship violin plot breaks naturalistic emotional naturalness logits down by relationship label.

Naturalistic emotional naturalness violins by relationship

The emotion scatter plot compares full-question and full-answer Arousal, Dominance, and Valence scores normalized to [-1, 1]. Dotted lines show per-dataset correlations with rho labels.

Question-answer Arousal, Dominance, and Valence scatter plots

STANCE

Stance Descriptions

stance	positive_original_%	negative_original_%	definition
Q0	82.1917808219178	17.80821917808219	Based on the audio, does the TARGET speaker sound warm and affiliative, or cold and detached?
Q1	48.24561403508772	51.75438596491228	Based on the audio, does the TARGET speaker sound compassionate and supportive, or callous and unsympathetic?
Q2	32.142857142857146	67.85714285714286	Based on the audio, does the TARGET speaker sound polite and respectful, or rude and disrespectful?
Q3	21.428571428571427	78.57142857142857	Based on the audio, does the TARGET speaker sound assertive and self-confident, or hesitant and inhibited?
Q4	57.84313725490196	42.15686274509804	Based on the audio, does the TARGET speaker sound straightforward and sincere, or sly and manipulative?
Q5	100.0	0.0	Based on the audio, does the TARGET speaker sound attentive and focused, or confused and distracted?
Q6	23.076923076923077	76.92307692307692	Based on the audio, does the TARGET speaker sound organized and goal-driven, or disorganized and unmotivated?
Q7	67.94871794871794	32.05128205128205	Based on the audio, does the TARGET speaker sound socially engaged and expressive with the other speaker, or withdrawn and disengaged?
Q8	45.070422535211264	54.929577464788736	Based on the audio, does the TARGET speaker accommodate and yield to the other speaker’s preferences, or do they try to control and dominate?
Q9	0.0	100.0	Based on the audio, does the TARGET speaker stay calm and avoid confrontation, or are they hostile and aggressive?