Persistent Memory in Multi-Agent LLM Inference: What It Costs, What It Buys, and When You Can Tell
Sep 30, 2026·
,,,·
0 min read
Hochan Son
Equal contribution
,Kyungdoe Han
Equal contribution
,Jaehan Koh
Xiaowu Dai
Wenlu Xu
Guang Cheng
Abstract
Decomposing long-context inference across cooperating agents bounds the key-value cache that each call holds, rather than the total evidence, which matters when cache memory is the binding limit. Many such systems add a persistent tier that stores and recalls reasoning traces, and an ablation that reports an accuracy gain usually supports that tier. We measure both on one three-tier agent architecture. Decomposition delivers a peak key-value working set of 14.3 MiB per query, against 35.5 and 35.3 MiB for the single-pass and retrieval-augmented baselines. The persistent tier does not. Across eight controlled dataset pairs at n=100 questions per arm, it costs +0.368 MiB [+0.167, +0.590] of peak cache and produces no detectable accuracy change (+0.015, 95% confidence interval [−0.011, +0.046]). We argue that this null result is structural. Single-question benchmarks supply each item with its own evidence and score it independently, and a correct measurement resets stored traces between conditions, so recall has nothing informative to retrieve. Reaching this result took four measurement corrections, and none was visible in the results table. Three inflated the apparent benefit, and the fourth made an effect of that size look resolvable. We give the conditions that an agent-memory ablation must satisfy, with detection procedures that need no knowledge of the specific defect.
Type
Publication
The 10th Machine Learning for Systems Workshop (MLForSys) at NeurIPS 2026, Sydney, Australia — poster
Status
Peer-reviewed

Authors
Hochan Son
(he/him)
Database Architect · SRE & DevOps Engineer · ML/GenAI Practitioner
Hochan Son is a distributed systems architect and site reliability engineer
who builds data infrastructure, ML pipelines, and high-scale distributed
systems, working the gap between GenAI/LLM research and the systems that run it
in production. His graduate research at the UCLA Trustworthy AI Lab spans
synthetic tabular data generation and agentic, intent-conditioned model
selection. His background spans Adtech, entertainment, and enterprise — from
MySpace and Hallmark Labs to Branch.io and ADP.