EMNLP 2026 Oral · Acoustic Memory in Audio LLMs

Why Can't They Remember?Uncovering Representation and Retrieval Bottlenecks in Multi-Turn Acoustic Memory

Audio language models remember what was said but forget what was heard. The sound is not erased; it drifts off the path the decoder reads from.

Yang Xiao1 Siyi Wang1 Han Yin2 Hong Jia3 Vidhyasaharan Sethu4 Eun-Jung Holden1 Ting Dang1
1The University of Melbourne 2KAIST 3The University of Auckland 4UNSW Sydney
arXiv PDF Film Explore the gap BibTeX
4,000evaluation instances
10ESC-50 sound classes
2–16dialogue turns (N)
8everyday topics
3open-weight LALMs
+12%relative gap Δ at N = 16
01

Overview

A spoken conversation carries more than words: rain, a car horn or a crying baby in the background. We ask whether audio LLMs keep that environmental context across turns, and if not, where it gets lost.

TL;DR

On EnvMem, acoustic memory decays faster than semantic memory as dialogues grow. Yet the sound remains linearly decodable from deep hidden states even on failed trials. Failures come from representational trajectory drift, not from erased information or misallocated attention, and patching in a clean representation rescues most of them.

RQ1

How large is the gap?

What is the performance gap between linguistic and environmental acoustic memory in multi-turn LALMs?

RQ2

Where does it come from?

Does acoustic memory failure stem from the representation level (information lost across layers) or the retrieval level (too little attention to early acoustic cues)?

Abstract

Large audio language models (LALMs) process both speech and environmental acoustic cues, yet struggle to retain non-speech information across multi-turn interactions. The performance gap between semantic (speech) and acoustic (non-speech) understanding remains poorly understood, and the underlying mechanisms of representation and retrieval are still unclear. This work introduces EnvMem, a controlled multi-turn benchmark designed to study this gap and identify the root causes of failures at the representation (i.e., latent embeddings) and retrieval levels (i.e., attention allocation). We further conduct post-hoc interventions to probe representational structure and attention dynamics. Our results reveal representational trajectory drift as the key failure mode, while showing that attention allocation tested here do not consistently explain the degradation. Overall, we provide a systematic framework for analyzing and improving non-linguistic memory in long-context LALMs, shedding light on future data and training design for robust acoustic memory modeling.

02

The EnvMem benchmark

The environmental sound appears only in the first user turn. Filler turns extend the context without new sounds, and the final turn asks about either the sound or a fact from that same first turn, so both probes share one audio context.

Overview of the EnvMem framework: (a) a multi-turn evaluation example where turn 1 contains speech mixed with a car horn and turn N asks an acoustic or a semantic multiple-choice question; (b) the curation pipeline of dialogue generation, keyword filtering, speech synthesis, environment sound sampling, audio mixing and dialogue assembly; (c) the topic distribution over eight everyday topics.
Overview of EnvMem. (a) A multi-turn example with the acoustic and semantic retrieval tasks. (b) The curation pipeline. (c) Topic distribution.
TURN 1 · user audio car horn honk · 10 dB SNR

“I need to finish the laundry later, and I wrote down 3 loads.”

TURNS 2 … N−1 · clean speech fillers

Same voice, same topic, keyword-filtered so no turn ever mentions the sound.

Acoustic probe TURN N

What was the background environment in the very first audio I sent?

  • (A) car_horn
  • (B) crying_baby
  • (C) rain
  • (D) dog
Semantic probe TURN N

When the user first mentioned the laundry, how many batches did it seem like?

  • (A) a single load
  • (B) one batch
  • (C) just a few
  • (D) several batches

Controlled by design

Sounds come from 10 ESC-50 classes in four families (weather/water, machine, vocal-bio, impulse), mixing continuous textures with transient events. Dialogues are GPT-4o templates over eight everyday topics; 38 of 208 candidate templates were removed by a 42-word lexicon that blocks textual leakage of the sound.

Every utterance is synthesized with one TTS voice (Kokoro af_heart, 16 kHz) to remove speaker confounds. 10 classes × 4 context lengths × 50 templates gives 2,000 acoustic and 2,000 semantic probes. Models: Qwen2.5-Omni, Kimi-Audio and Qwen2-Audio, all with white-box access.

Acoustic classes in EnvMem. Each class has 50 templates per context length, 200 samples in total.
Sound familyESC-50 classes
Weather / Waterrain, thunderstorm, sea waves
Machineengine, vacuum cleaner, car horn
Vocal-Biodog, crying baby
Impulseclock tick, crackling fire
03

Sound fades faster than words

Pick a model and a dialogue length. Bars show accuracy on the two probes; the thin mark is the model's own single-turn (N = 2) baseline. Δ compares the relative drop of acoustic vs semantic memory, so a weak initial perception is factored out.

Dialogue

anchor turn with the soundclean filler turnsprobe question

Accuracy (4-way multiple choice)

Semantic
Acoustic
Relative gap Δ(N)

Accuracies are read from the plot below (rounded); Δ values are those printed in the paper. Δ(N) = [acca(2) − acca(N)] / acca(2) − [accs(2) − accs(N)] / accs(2); Δ > 0 means acoustic memory degrades faster.

Three line plots of accuracy versus number of turns N for Qwen2.5-Omni, Kimi-Audio and Qwen2-Audio. Semantic memory stays higher than acoustic memory, and the relative gap reaches +12%, +8% and +8% at N = 16.
Acoustic vs semantic memory over increasing dialogue turns. At N = 16 the relative gap is +12% for Qwen2.5-Omni and +8% for both Kimi-Audio and Qwen2-Audio. Qwen2-Audio starts with a perception gap (0.39 acoustic vs 0.90 semantic at N = 2), which is why its Δ is negative at short lengths.
04

Diagnosing the failure

Two hypotheses could explain the gap. Either the sound is lost as hidden states propagate (representation), or it is kept but the model fails to attend to it (retrieval). We test both observationally, then causally. Main analyses use Qwen2.5-Omni.

1

Not erasedThe sound stays decodable in deep layers, even when the model answers wrong.

A linear probe on the final query token recovers the 10-way sound class from layer 20 onward. At N = 16 it reaches about 70% by layer 25 (about 80% at N = 2). Restricted to trials the model failed, layer-25 probe accuracy is still about 48%, far above the 10% chance level.

Context length also moves where the sound is integrated: at N = 2 the probe reaches about 0.50 by layer 5, while at N = 8 and 16 mid-layer accuracy stays near 0.20 up to layer 15 and jumps in layers 16–21.

Layer-wise linear probe accuracy for N = 2, 4, 8 and 16 on all samples, rising sharply after layer 15 and peaking near layer 25.
(a) all samples
Layer-wise linear probe accuracy restricted to failed trials, still reaching about 0.5 in deep layers for N = 16.
(b) failed samples only
2

DriftFailed trajectories collapse toward a short-context regime.

Cross-layer CKA compares hidden states of failed long-context trials with successful trials of the same N and with short (N = 2) trials, matched on sound class. Layers 0–5 show no differentiation. In layers 6–19, failed trials are consistently closer to the short baseline than to successful ones: δ = CKA(failed, short) − CKA(failed, success) is 0.054, 0.055 and 0.067 at N = 4, 8, 16. In layers 20–28 a high-similarity block appears but peaks off the diagonal, a phase-shifted integration.

We call this representational trajectory drift: the information is present, but its format no longer matches what the decoding pathway expects.

Four cross-layer CKA heatmaps: failed versus success and failed versus clean N = 2, for N = 4 and N = 16. Similarity is higher against the short-context baseline, with an off-diagonal block in deep layers.
Cross-layer CKA. Top: failed vs same-N success. Bottom: failed vs short N = 2.
CKA scanlines for Qwen2.5-Omni across N, showing failed trials aligning more with the short-context baseline in the middle layers.
CKA scanlines for Qwen2.5-Omni (appendix).
3

Not attentionAttention allocation does not explain the failure.

If long contexts diluted attention, the anchor turn should receive less attention as N grows. Instead, at N = 4 the anchor gets less attention than fillers through most layers, while at N = 16 it gets more (+0.005 to +0.010). Failed trials are also no more diffuse than successful ones: the turns needed to cover 90% of attention differ by only +0.04, −0.10 and −0.10.

Anchor minus filler attention mass by layer for N = 4 (mostly negative) and N = 16 (mostly positive).
Anchor attention gap by layer. Negative means less attention to the anchor than to fillers.
Attention concentration (cov90) on class-matched failed vs successful samples.
MetricNFailedSuccessΔd
cov90 (turns)42.712.67+0.04
86.156.25−0.10
1610.1510.25−0.10
cov90 (tokens)464.862.5+2.4
896.694.8+1.8
16136.5130.6+5.9
4

CausalRestoring the representation rescues failed predictions; editing attention does not.

Activation patching. On failed trials we overwrite the decoding-position hidden state at layer 25 with a donor vector. For Qwen2.5-Omni at N = 16, a clean same-class donor lifts accuracy from 0.13 to 0.75, while a wrong-class donor (0.17) and the model's own contaminated vector (0.22) stay near baseline. Qwen2-Audio shows the same pattern (clean donors 0.30–0.45).

Attention masking. Amplifying the anchor turn, suppressing fillers, or a magnitude-matched random boost all change accuracy within noise; every 95% CI includes zero. A sweep over layer offsets and boost sizes peaks at +3.7, against +1.3 for the random control.

Grouped bars of rescued accuracy on failed targets for Qwen2.5-Omni and Qwen2-Audio at N = 4, 8, 16: only the clean same-class donor rises well above baseline.
Activation patching on failed targets. Only clean same-class donors rescue failed predictions, ruling out a generic perturbation effect.
Attention-mask interventions on the failed subset (Qwen2.5-Omni, 80 samples per N). Baseline is accuracy (%); intervention rows are Δ vs baseline with paired-bootstrap 95% CIs.
N = 4N = 8N = 16
Baseline (%)12.5 [6.2, 20.0]12.5 [6.2, 20.0]15.0 [7.5, 22.5]
Anchor amplification+1.3 [−5.0, +7.5]+1.3 [−5.0, +7.5]+1.2 [−6.2, +8.8]
Filler suppression0.0 [−6.2, +7.5]+2.5 [−5.0, +11.2]0.0 [−7.5, +7.5]
Random control−1.3 [−7.5, +5.0]0.0 [−6.2, +7.5]+1.2 [−6.2, +8.8]
05

Robustness

The effect is not tied to one mixing level or one model.

Layer-wise probe accuracy for Kimi-Audio, with delayed recovery in deep layers for long contexts.
Kimi-Audio probe
Layer-wise probe accuracy for Qwen2-Audio, with delayed recovery in deep layers for long contexts.
Qwen2-Audio probe

Across models. Kimi-Audio and Qwen2-Audio show the same delayed acoustic integration at N ≥ 8: the sound is barely decodable in early and middle layers, then recovers sharply in the deepest blocks.

Across SNR (Qwen2.5-Omni, N = 16)

Both the behavioural gap Δ and the Phase-II alignment shift δ persist at 5 and 20 dB (500-instance subset).
SNRΔ at N = 16Phase-II δ
5 dB+9%0.045
10 dB (default)+12%0.067
20 dB+15%0.070
06

What this means

Acoustic information is largely preserved but becomes functionally misaligned with decoding in long contexts.

Benchmarks

Separate sound from speech

Holding scripts, voice and turn structure fixed while only the probe target changes isolates modality-specific memory from content recall.

Bottleneck

Representation, not retrieval

Pre- and post-training that under-emphasize acoustic events, and weakly couple them with semantics, likely drive the gap.

Mitigation

Train for format-compatible memory

Post-hoc patching shows the failure is correctable; the next step is integrating acoustic representations during training.

07

BibTeX

If you find EnvMem useful, please cite:

@inproceedings{xiao2026envmem,
  title     = {Why Can't They Remember? Uncovering Representation and Retrieval
               Bottlenecks in Multi-Turn Acoustic Memory},
  author    = {Xiao, Yang and Wang, Siyi and Yin, Han and Jia, Hong and
               Sethu, Vidhyasaharan and Holden, Eun-Jung and Dang, Ting},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods
               in Natural Language Processing (EMNLP)},
  year      = {2026}
}