Skip to content

Who Remembers What? Multi-Granularity Memory Extraction for Multi-User, Multi-Agent Interaction

Yunzhe Wang Mesut Erhan Unal Sheng Zhang Shabnam (Shana) Ghadar Peng Tang

AWS Agentic AI

PaperarXiv
TL;DR Evaluated with 5 users and 7 agents

Shared learning. Scoped memory.

MixUA turns experience from individual user-agent interactions into lessons others can reuse. Each lesson goes to the audience it can help, within the limits of what may be shared.

Four scopes PAIR USER AGENT GLOBAL
up to
16pts
pass-rate gain over no memory
up to
5pts
over a single global memory
up to
90%
of stated preferences retained
at most
0.02%
of privacy probes find a leaked private fact
Abstract

Self-evolving agents improve their own skills, memory, and harness code from experience without retraining the base LLM, and group-evolving agents go further by sharing these lessons across agents. Yet most such systems overlook that sharing across users and heterogeneous agents requires deciding who can benefit from each lesson and what information must remain private. We present MixUA, a memory system that distills independent user-agent trajectories into lessons and assigns each lesson to a user-agent pair, user, agent, or global scope according to whom it benefits and what it may disclose. For evaluation, we introduce CrossLearnBench, a mixed-user-agent benchmark that pairs established agent tasks with user preferences and privacy probes that test whether private facts leak into memory other users can read. With five users and seven agents, MixUA raises pass rates by up to 16 points over no memory and 5 over a single global memory, even on tasks previously tried only by other users on other agents. It retains up to 90% of previously stated preferences, and at most 0.02% of privacy probes find a planted private fact in a forbidden scope. We hope MixUA offers a scalable path toward a global system where users and agents co-evolve.

Who should remember each lesson?

A scope defines who may read a lesson. Is it specific to one user, one agent, both, or neither? Here, an agent is a particular combination of an LLM and its tools.

PAIR
One user, working with one particular agent. Stores lessons specific to that combination. For example, a user may want shorter replies from a verbose agent but more detail from a terse one.
USER
One user, across all agents. Stores personal preferences and private context that should follow the user when they switch agents, such as their preferred document format. Other users cannot read it.
AGENT
One agent, across all users. Stores guidance tied to that agent's model or tools, such as how to avoid a recurring tool-calling mistake. It can help anyone using that agent, without sharing private user information.
GLOBAL
All users, across all agents. Stores reusable knowledge that does not depend on a particular user or agent, such as a library bug and its fix. It must be safe to share across users.

Lessons from private details land only in the user's own USER or PAIR memory.

Rows are users, columns are agents. The outlined tile is the pair that learned the lesson.

How it works

Independent user-agent pairs produce trajectories that are verified, filtered into a reusable and a protected stream, and written into a shared memory store with GLOBAL, USER, AGENT and PAIR sections. A periodic store review re-scopes and merges entries, and relevant memory is retrieved for each authorized audience.

Swipe to see the whole figure.

  1. An LLM splits lessons from a single trajectory into shareable and private parts.

  2. Each lesson is written to the scope it fits; private ones stay with their user.

  3. Before each task, a pair reads only what it is allowed to see, most relevant first.

Results

Scatter of utility gain over no memory against preference retention. PAIR-only retains preferences but gains little; GLOBAL-only gains but retains few; the four MixUA variants sit together in the top-right corner.

Scheduled tasks

Only MixUA gets both

Single-scope memories trade task success for preferences; MixUA gains 13-16 points while retaining 80-90%.

Cumulative pass rate over eight rounds of random tasks: MixUA rises highest, GLOBAL-only next, with PAIR-only and no memory lower.

Random tasks, 8 rounds

It helps on new tasks

On tasks no pair had tried before, MixUA gains 10.1 points, nearly twice the 5.5 of a single global memory.

Preference retention of users told in round 1, at 5, 20 and 100 users: MixUA stays highest, PAIR-only falls as users are added, and GLOBAL-only and no memory stay low.
  • No memory
  • PAIR only
  • GLOBAL only
  • MixUA

5 to 100 users

It holds at 100 users

MixUA still retains 78% of stated preferences; a single global memory retains 11%.

CrossLearnBench

Established agent tasks, plus synthetic users with planted preferences and private facts.

Tasks from
Run by
seven LLM-harness combinations
Users bring
preferences private facts

BibTeX

arXiv preprint 
@misc{wang2026whoremembers,
  title         = {Who Remembers What? Multi-Granularity Memory Extraction for Multi-User, Multi-Agent Interaction},
  author        = {Wang, Yunzhe and Unal, Erhan and Zhang, Sheng and Ghadar, Shabnam and Tang, Peng},
  year          = {2026},
  eprint        = {2611.10439},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2611.10439}
}