Persistent Memory in Multi-Agent LLM Inference: What It Costs, What It Buys, and When You Can Tell

Sep 30, 2026·
Hochan Son
Hochan Son
Equal contribution
,
Kyungdoe Han
Equal contribution
,
Jaehan Koh
,
Xiaowu Dai
,
Wenlu Xu
,
Guang Cheng
· 0 min read
Abstract
Decomposing long-context inference across cooperating agents bounds the key-value cache that each call holds, rather than the total evidence, which matters when cache memory is the binding limit. Many such systems add a persistent tier that stores and recalls reasoning traces, and an ablation that reports an accuracy gain usually supports that tier. We measure both on one three-tier agent architecture. Decomposition delivers a peak key-value working set of 14.3 MiB per query, against 35.5 and 35.3 MiB for the single-pass and retrieval-augmented baselines. The persistent tier does not. Across eight controlled dataset pairs at n=100 questions per arm, it costs +0.368 MiB [+0.167, +0.590] of peak cache and produces no detectable accuracy change (+0.015, 95% confidence interval [−0.011, +0.046]). We argue that this null result is structural. Single-question benchmarks supply each item with its own evidence and score it independently, and a correct measurement resets stored traces between conditions, so recall has nothing informative to retrieve. Reaching this result took four measurement corrections, and none was visible in the results table. Three inflated the apparent benefit, and the fourth made an effect of that size look resolvable. We give the conditions that an agent-memory ablation must satisfy, with detection procedures that need no knowledge of the specific defect.
Type
Publication
The 10th Machine Learning for Systems Workshop (MLForSys) at NeurIPS 2026, Sydney, Australia — poster
Status
Peer-reviewed
publications
Hochan Son
Authors
Hochan Son (he/him)
Database Architect · SRE & DevOps Engineer · ML/GenAI Practitioner
Hochan Son is a distributed systems architect and site reliability engineer who builds data infrastructure, ML pipelines, and high-scale distributed systems, working the gap between GenAI/LLM research and the systems that run it in production. His graduate research at the UCLA Trustworthy AI Lab spans synthetic tabular data generation and agentic, intent-conditioned model selection. His background spans Adtech, entertainment, and enterprise — from MySpace and Hallmark Labs to Branch.io and ADP.