SPEARBench Detailed Report

Summary paragraph

gpt-realtime-2 performs well on intelligibility, with substantially lower CER/WER than the original answers on both improvised and naturalistic subsets, and it also keeps interruptions low and generally comparable to or better than the originals. The main tradeoff is temporal: responses are much longer in duration than the source answers, even though latency is much shorter. Language ID is highly stable, with the model’s answers effectively all English, and dialect behavior shows moderate entrainment with high variance, suggesting some style adaptation without collapsing to a narrow dialect. On stance, it largely preserves the original sign of social and interactional cues, with only small rates of positive or negative sign flips, indicating strong high-level consistency in perceived attitude. The report mentions that emotional naturalness and explainable speech features were analyzed, but it does not provide strong textual conclusions for those signals in this summary. Overall, the model’s profile is strong transcription accuracy and conversational stability, balanced against notably longer spoken outputs.

Data, Models, and Setup

Data

metric	Improvised dev	Improvised test	Naturalistic dev	Naturalistic test	Total
Unique original dialogues	270	181	480	511	1442
Selected dialogues	1165	994	1529	1731	5419
Hours of questions	4.78h	3.76h	6.76h	7.47h	22.77h
Hours of original answers	2.67h	2.29h	5.23h	4.36h	14.56h
Original total hours	7.45h	6.05h	12.00h	11.83h	37.33h
Hours of generated answers	6.97h	5.96h	9.21h	10.30h	32.44h
Mean original answer length	8.26s	8.28s	12.32s	9.07s
Mean generated answer length	19.20s	19.37s	19.91s	19.30s

Models

parameter	value
LLM model used for inference	gpt-realtime-2
ASR models used	Qwen3-ASR-0.6B, whisper-large-v3
Language ID model	facebook/mms-lid-126
Dialect ID model	tiantiaf/voxlect-english-dialect-whisper-large-v3
LLM used for stance	gpt-audio-1.5
Statistical tests used for p-values	Mann-Whitney U Test
UTMOS model	tarepan/SpeechMOS:v1.2.0 utmos22_strong
VAD model	silero-vad

Setup

parameter	value
protocol	seamless_2t_2s_questions
selection_method	end_with_question
min_turns	2
min_speakers	2
splits	dev,test
subsets	improvised,naturalistic
data_root	data/seamless_2t_2s_questions
results_root	results/seamless_2t_2s_questions

Intelligibility and Interruption Metrics

Metric	System	model_mean_naturalistic	original_mean_naturalistic	model_mean_improvised	original_mean_improvised
CER	Qwen3-ASR-0.6B	6.19	19.70	4.04	16.27
CER	whisper-large-v3	<NA>	<NA>	6.94	13.99
WER	Qwen3-ASR-0.6B	15.30	32.24	13.39	28.28
WER	whisper-large-v3	<NA>	<NA>	15.75	20.11
UTMOS		4.2670	2.1655	4.2882	2.3207
latency		107.8084	929.6360	97.0190	415.3924
average number of interruptions per dialogues		0.58	0.50	0.22	0.55
interrupted time (s)		0.548	0.520	0.228	0.522
number of dialogues with interruption		578/1921	545/1731	136/1107	377/994

Intelligibility and interruption metric histograms with KDE

Basic metric histogram gallery

Turn Taking Surprisal

subset	metric	model_value	original_value	mean_diff	std_diff	p_value	n
improvised	mean_nll	2.990628377585879	3.392070671418477	-0.4014	1.7257	8.780e-05	509
improvised	tail_nll	3.0128331562686186	3.520741825309412	-0.5079	1.8835	8.343e-05	509
improvised	dialog_nll	3.0017307669272486	3.4564062483639444	-0.4547	1.7976	7.421e-05	509
improvised	naturalness_score	3.0017307669272486	3.4564062483639444	-0.4547	1.7976	7.421e-05	509
naturalistic	mean_nll	3.0499898119100917	3.9220984000715045	-0.8721	1.6417	9.433e-65	882
naturalistic	tail_nll	3.0704471376335114	4.1670074320954695	-1.0966	1.8474	6.019e-66	882
naturalistic	dialog_nll	3.0602184747718013	4.044552916083487	-0.9843	1.7330	5.190e-66	882
naturalistic	naturalness_score	3.0602184747718013	4.044552916083487	-0.9843	1.7330	5.190e-66	882

Language and Dialect ID

subset	% Eng in models' answers	% Eng in original answers	second most spoken language in original answers	percentage 2nd language in original answers
improvised	100.0	99.73545	Vietnamese	0.088183
naturalistic	100.0	99.01332	dan	0.345338

Subset	Question/Answer	East Asia	English	Germanic	Irish	North America	Oceania	Other	Romance	Scottish	Semitic	Slavic	South African	Southeast Asia	South Asia
improvised	Question	0.347826	0.086957	0.086957	5.826087	83.391304	0.086957	0.000000	9.043478	0.086957	0.086957	0.347826	0.000000	0.434783	0.086957
improvised	Answer	0.000000	0.000000	0.000000	1.827676	97.476066	0.000000	0.000000	0.435161	0.000000	0.000000	0.000000	0.000000	0.261097	0.000000
naturalistic	Question	1.904297	0.585938	0.292969	4.638672	72.558594	0.244141	0.195312	16.748047	0.146484	0.292969	0.244141	0.341797	1.269531	0.341797
naturalistic	Answer	0.000000	0.000000	0.000000	1.272016	98.336595	0.048924	0.000000	0.097847	0.097847	0.000000	0.000000	0.000000	0.146771	0.000000

Dialectal Metrics

subset	metric	mean_diff	n	detail
improvised	Dialectal entrainment	0.4186	976	computed from dialect_logits.csv question_log_logits and answer_log_logits
improvised	Dialectal variance	89.3485	976	computed from dialect_logits.csv question_log_logits and answer_log_logits
naturalistic	Dialectal entrainment	0.4077	1685	computed from dialect_logits.csv question_log_logits and answer_log_logits
naturalistic	Dialectal variance	90.0412	1685	computed from dialect_logits.csv question_log_logits and answer_log_logits

Dialect question-to-answer confusion matrix

Dialect score spider profiles for question and answer fields

Emotional Naturalness

The relationship violin plot breaks naturalistic emotional naturalness logits down by relationship label.

Naturalistic emotional naturalness violins by relationship

The emotion scatter plot compares full-question and full-answer Arousal, Dominance, and Valence scores normalized to [-1, 1]. Dotted lines show per-dataset correlations with rho labels.

Question-answer Arousal, Dominance, and Valence scatter plots

STANCE

Stance Descriptions

stance	positive_original_%	negative_original_%	definition
Q0	82.1917808219178	17.80821917808219	Based on the audio, does the TARGET speaker sound warm and affiliative, or cold and detached?
Q1	48.24561403508772	51.75438596491228	Based on the audio, does the TARGET speaker sound compassionate and supportive, or callous and unsympathetic?
Q2	32.142857142857146	67.85714285714286	Based on the audio, does the TARGET speaker sound polite and respectful, or rude and disrespectful?
Q3	21.428571428571427	78.57142857142857	Based on the audio, does the TARGET speaker sound assertive and self-confident, or hesitant and inhibited?
Q4	57.84313725490196	42.15686274509804	Based on the audio, does the TARGET speaker sound straightforward and sincere, or sly and manipulative?
Q5	100.0	0.0	Based on the audio, does the TARGET speaker sound attentive and focused, or confused and distracted?
Q6	23.076923076923077	76.92307692307692	Based on the audio, does the TARGET speaker sound organized and goal-driven, or disorganized and unmotivated?
Q7	67.94871794871794	32.05128205128205	Based on the audio, does the TARGET speaker sound socially engaged and expressive with the other speaker, or withdrawn and disengaged?
Q8	45.714285714285715	54.285714285714285	Based on the audio, does the TARGET speaker accommodate and yield to the other speaker’s preferences, or do they try to control and dominate?
Q9	0.0	100.0	Based on the audio, does the TARGET speaker stay calm and avoid confrontation, or are they hostile and aggressive?