SPEARBench Detailed Report - gemini-2.5-flash-native-audio-preview

Summary paragraph

For the seamless_2t_2s_questions protocol, gemini-2.5-flash-native-audio-preview shows strong language control, with answers staying overwhelmingly English and only tiny secondary-language spillover in both improvised and naturalistic data. Its turn-taking is broadly human-like but still less natural than the human baseline, because mean, tail, and dialog surprisal are consistently higher than human values across both subsets, while interruption metrics show essentially no interruptions for the model versus frequent interruptions in human dialogue. Dialect behavior indicates modest entrainment with nontrivial variability, though naturalistic interactions show slightly lower entrainment and slightly higher dialect variance than improvised ones. On stance, the model largely preserves the human sign pattern but is not fully aligned, with about 95.9% same-sign agreement and a small tendency to shift interpretations more positive or more negative than humans. The report also does not provide intelligibility/interruption findings beyond the interruption counts or any concrete explainable-feature conclusions, so those signals cannot be used to assess additional strengths or weaknesses here. Overall, the main tradeoff is very stable English speech and broadly human-like stance/dialect behavior versus a measurable gap in turn-taking naturalness and some sensitivity to dialect and affective interpretation.

Data, Models, and Setup

Data

metric	Improvised dev	Improvised test	Naturalistic dev	Naturalistic test	Total
Unique human dialogues	270	181	480	511	1442
Selected dialogues	1165	994	1529	1731	5419
Hours of questions	4.78h	3.76h	6.76h	7.47h	22.77h
Hours of human answers	2.67h	2.29h	5.23h	4.36h	14.56h
Human total hours	7.45h	6.05h	12.00h	11.83h	37.33h
Hours of generated answers	2.06h	1.79h	3.12h	3.21h	10.19h
Mean human answer length	8.26s	8.28s	12.32s	9.07s
Mean generated answer length	6.40s	6.52s	7.40s	6.72s

Models

parameter	value
LLM model used for inference	gemini-2.5-flash-native-audio-preview
ASR models used	Qwen3-ASR-0.6B, whisper-large-v3
Language ID model	facebook/mms-lid-126
Dialect ID model	tiantiaf/voxlect-english-dialect-whisper-large-v3
LLM used for stance	gpt-audio-1.5
Statistical tests used for p-values	Mann-Whitney U Test
UTMOS model	tarepan/SpeechMOS:v1.2.0 utmos22_strong
VAD model	silero-vad

Setup

parameter	value
protocol	seamless_2t_2s_questions
selection_method	end_with_question
min_turns	2
min_speakers	2
splits	dev,test
subsets	improvised,naturalistic
data_root	data/seamless_2t_2s_questions
results_root	results/seamless_2t_2s_questions

Intelligibility and Interruption Metrics

Metric	System	model_mean_naturalistic	human_mean_naturalistic	model_mean_improvised	human_mean_improvised
CER	Qwen3-ASR-0.6B	78.39	19.70	77.82	16.27
CER	whisper-large-v3	78.33	17.13	77.74	13.99
WER	Qwen3-ASR-0.6B	76.99	32.24	76.30	28.28
WER	whisper-large-v3	76.60	25.42	75.78	20.11
UTMOS		3.6846	2.1655	3.7837	2.3207
latency		178.4428	929.6360	152.2267	415.3924
average number of interruptions per dialogues		0.00	0.50	0.00	0.55
interrupted time (s)		0.000	0.520	0.000	0.522
number of dialogues with interruption		0/1721	545/1731	0/988	377/994

Intelligibility and interruption metric histograms with KDE

Basic metric histogram gallery

Turn Taking Surprisal

metric	model - improvised	model - naturalistic	human - improvised	human - naturalistic
mean_nll	4.797	4.688	4.439	4.525
tail_nll	5.86	5.917	5.286	5.598
dialog_nll	5.328	5.303	4.862	5.061
naturalness_score	5.328	5.303	4.862	5.061

Language and Dialect ID

subset	% Eng in models' answers	% Eng in human answers	second most spoken language in models' answers	percentage 2nd language in models' answers	second most spoken language in human answers	percentage 2nd language in human answers
improvised	99.896587	99.73545	mlg	0.103413	Vietnamese	0.088183
naturalistic	99.527745	99.01332	Japanese	0.177096	dan	0.345338

Subset	Question/Answer	East Asia	English	Germanic	Irish	North America	Oceania	Other	Romance	Scottish	Semitic	Slavic	South African	Southeast Asia	South Asia
improvised	Question	0.404858	0.000000	0.101215	5.870445	80.769231	0.101215	0.000000	9.412955	0.101215	0.101215	0.303644	0.000000	0.506073	0.101215
improvised	Answer	0.414079	0.000000	0.000000	0.103520	95.548654	0.000000	0.310559	0.414079	0.000000	0.000000	0.000000	0.000000	3.209110	0.000000
naturalistic	Question	2.033701	0.464846	0.348635	4.532249	71.237653	0.232423	0.174317	16.211505	0.174317	0.348635	0.174317	0.290529	1.394538	0.348635
naturalistic	Answer	0.296560	0.059312	0.000000	0.118624	96.144721	0.177936	0.118624	0.415184	0.000000	0.000000	0.000000	0.000000	2.609727	0.059312

Dialectal Metrics

subset	metric	mean_diff	n	detail
improvised	Dialectal entrainment	0.5313	948	computed from dialect_logits.csv question_log_logits and answer_log_logits
improvised	Dialectal variance	123.7753	948	computed from dialect_logits.csv question_log_logits and answer_log_logits
naturalistic	Dialectal entrainment	0.4981	1644	computed from dialect_logits.csv question_log_logits and answer_log_logits
naturalistic	Dialectal variance	127.7142	1644	computed from dialect_logits.csv question_log_logits and answer_log_logits

Dialect question-to-answer confusion matrix

Dialect score spider profiles for question and answer fields

Emotional Naturalness

The relationship violin plot breaks naturalistic emotional naturalness logits down by relationship label.

Naturalistic emotional naturalness violins by relationship

The emotion scatter plot compares full-question and full-answer Arousal, Dominance, and Valence scores normalized to [-1, 1]. Dotted lines show per-dataset correlations with rho labels.

Question-answer Arousal, Dominance, and Valence scatter plots

STANCE

Stance Descriptions

stance	positive_human_%	negative_human_%	definition
Q0	81.9672131147541	18.0327868852459	Based on the audio, does the TARGET speaker sound warm and affiliative, or cold and detached?
Q1	45.744680851063826	54.255319148936174	Based on the audio, does the TARGET speaker sound compassionate and supportive, or callous and unsympathetic?
Q2	29.78723404255319	70.2127659574468	Based on the audio, does the TARGET speaker sound polite and respectful, or rude and disrespectful?
Q3	21.0	79.0	Based on the audio, does the TARGET speaker sound assertive and self-confident, or hesitant and inhibited?
Q4	57.142857142857146	42.857142857142854	Based on the audio, does the TARGET speaker sound straightforward and sincere, or sly and manipulative?
Q5	100.0	0.0	Based on the audio, does the TARGET speaker sound attentive and focused, or confused and distracted?
Q6	21.666666666666668	78.33333333333333	Based on the audio, does the TARGET speaker sound organized and goal-driven, or disorganized and unmotivated?
Q7	68.1159420289855	31.884057971014492	Based on the audio, does the TARGET speaker sound socially engaged and expressive with the other speaker, or withdrawn and disengaged?
Q8	50.0	50.0	Based on the audio, does the TARGET speaker accommodate and yield to the other speaker’s preferences, or do they try to control and dominate?
Q9	0.0	100.0	Based on the audio, does the TARGET speaker stay calm and avoid confrontation, or are they hostile and aggressive?