Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory

We want to make a claim:

Agent memory cannot be retrieve-only.

Retrieve-only memory is closer to a notebook than to human memory.

A notebook may store information perfectly, but it helps only when you realize that you should look something up. Human memory is different: important facts can continuously shape our judgment, and world knowledge connects things that appear unrelated on the surface.

Yet most agent-memory systems today are notebook-like. They write user information into an external store, then retrieve “relevant” memories when a new query arrives. Whether the implementation is a vector database, a knowledge graph, or agentic RAG, the basic order is usually the same:

First decide what is relevant from the query.
Then let the model reason.

That order creates a structural blind spot.

A user says:

“I’m allergic to tree nuts.”

Later, they ask:

“I want to try making macarons. Any recipes?”

A good agent should warn them that traditional macarons use almond flour. But many memory systems simply provide a recipe: “tree-nut allergy” and “macarons” are not similar enough for the allergy memory to be retrieved in the first place.

Or consider this:

“I have lilies at home.”
“I want to adopt a cat.”

A good agent should warn that lilies are highly toxic to cats and should be removed first. But a retrieve-only system may just give generic advice about caring for a new cat, because “lilies” and “cats” have almost no surface-level similarity.

The troubling part is that these systems may still answer direct questions perfectly:

“What am I allergic to?”
“What plants do I have at home?”

So users are led to believe that the agent truly remembers—and will “keep in mind”—their important information.

But this can be a false promise: the memory exists in the database, yet fails to affect the agent’s behavior when it actually matters.

We tested representative state-of-the-art memory systems, including HippoRAG 2, A-RAG, A-Mem, Mem0, MemoryOS, and xMemory. Across this class of cases, they show the same systematic failure pattern: they can recall the fact on demand, but fail to apply it when a world-knowledge bridge is needed.

To measure this phenomenon, we introduce InMind Memory Bench, a benchmark for the implicit-association blind spot in agent memory.

Retrieval is necessary. But retrieval alone is not enough.

A reliable agent memory needs to answer a harder question: not only what should be retrieved now?, but also what information must remain available because it could matter in ways the current query does not reveal?