When Graph Retrieval Fails to Transfer

The graph behaved as designed. That was precisely why its failure to transfer deserved attention: the design fit the first test unusually well.

On a deterministic synthetic memory corpus, a typed relation retriever raised multi-hop recall@10 from 0.325 for BM25 to 0.800. The gain was large, reproducible, and protected by a regression gate. At that point, graph-structured retrieval looked like a credible approach to long-term agent memory.

I then moved the experiment to real dialogue. On a pooled subset of LongMemEval containing 20 multi-session questions and 9,697 dialogue turns, the graph variants scored 0.168 recall@10, below BM25 at 0.193. A dense retriever reached 0.291; a later coarse-to-fine search reached 0.438. The synthetic advantage was absent, and the graph also scored below the baseline in this small experiment.

This note reconstructs that reversal. It follows the tests in the order I ran them, then asks which differences between the two regimes could account for the result and which conclusions the evidence does not support.

The question

Long-term conversational memory poses a familiar retrieval problem. A question may depend on two statements separated by weeks of interaction:

1. Anna introduced me to Marcus at a meetup.
2. The person I met at the meetup recommended Dune.

Answering requires a join. Lexical retrieval may return either the introduction or the recommendation without connecting them. A graph supplies an appealing representation: people and venues become nodes, introductions and recommendations become typed edges, and traversal recovers the evidence chain.

That intuition leads to a deliberately narrow research question:

Does a typed relation graph improve the retrieval of multi-session evidence in long-term dialogue, or can its apparent advantage be an artifact of the benchmark and extractor that define the graph?

The outcome here is retrieval, not final answer quality from a reader LLM. Nor does the experiment evaluate graph memory as a provenance or maintenance substrate. Graphs may remain useful for either purpose even when they are poor rankers.

What was measured

For a question q, let G_q be its gold evidence turns and T₁₀(q) the first ten retrieved turns. The reported metric is macro-averaged evidence recall:

Macro-averaging matters because a multi-session question may require several evidence turns. Recovering one supporting statement is not equivalent to recovering the evidence set.

I did not begin with a single benchmark run. The retrieval design moved through four tests, each prompted by a limitation exposed in the preceding one.

Round 1: reachability without ranking

The first graph connected entries through co-mentioned people and venues, then fused its candidates with BM25. On the synthetic V2 corpus—seed 42, five simulated years, approximately 11,500 entries—it tied BM25 at 0.325 multi-hop recall@10. The graph walk alone scored 0.175.

That result separated two properties I had initially treated together. Co-occurrence made many entries reachable, but a path was not a relevance score. Popular people and repeated venues flooded the candidate set; once noisy graph candidates entered fusion, they could displace useful lexical hits.

Round 2: the typed-relation gain

I next replaced generic co-mention with explicit relations. One edge represented an introduction, $P \xrightarrow{\text{introduced at } V} Q$; another represented a recommendation, $Q \xrightarrow{\text{recommended}} B$. For coreferential cases, the retriever could bridge through a venue and favor recommendations close in time to the introduction. Graph expansion ran only when a question contained introduction or recommendation cues, leaving ordinary single-hop queries on the BM25 path.

This version scored 0.800 recall@10, compared with 0.325 for BM25, and 0.900 versus 0.650 on the repository’s answer-in-top-10 proxy. At a larger synthetic scale of approximately 29,600 entries, the absolute scores fell, but the relative retrieval advantage remained: 0.422 versus 0.172.

The predefined synthetic gate passed. Yet the success came with an important coupling. The system and generator shared a vocabulary and a causal grammar: a small family of introduction and recommendation templates generated the corpus, while the retriever searched for a corresponding family of cues. It also built a gazetteer from the synthetic ground-truth entity names. That coupling does not invalidate the experiment, but it narrows the scope of the result.

Round 3: the oracle warning

Before paying for live extraction, I replayed a ground-truth oracle through the same artifact interface intended for an LLM. The oracle tier changed synthetic multi-hop recall by −0.125 at the default scale and +0.031 at the larger scale. At first, this was surprising.

The oracle resolved ambiguous recommendations to person names. Reused names then created new cross-chain collisions, whereas the deterministic venue-and-time bridge had already avoided some of them. Better extraction did not necessarily produce better ranking. In retrospect, this was an early warning: graph quality, entity quality, and retrieval quality were not interchangeable quantities.

Round 4: real dialogue

For the real-text campaign, I used two public conversational-memory benchmarks. LoCoMo contains long conversations spanning as many as 35 sessions. On the evaluated sample, BM25 already reached 0.784 recall@10 on category-2 multi-hop questions. With little unmet retrieval gap for the graph to close, it underperformed BM25 across the tested categories.

LongMemEval offered the more relevant stress test because its multi-session questions explicitly require evidence synthesis across sessions. I pooled every turn from 20 scoreable multi-session instances into one 9,697-entry search space and evaluated all retrievers at k = 10.

Both graph variants were 0.025 below BM25. Per-instance evaluation pointed in the same direction—0.360 and 0.363 for the two graph variants versus 0.372 for BM25—but the small sample produced an estimated uncertainty much larger than the observed difference. The defensible interpretation is therefore narrow: the synthetic gain did not reproduce, and this sample provided no evidence that the graph improved retrieval.

Why the result changed

The experiments do not identify a controlled causal explanation. They do, however, expose several points at which the synthetic task and real dialogue place different demands on a retriever. I treat these as diagnostics to test, not as established causes of failure.

A schema that already knew the task

In the synthetic corpus, the target chains were generated from a constrained set of introduction–recommendation templates. The deterministic retriever encoded the same relations and cues. The benchmark could therefore test whether the implementation recovered planted joins amid distractors, but it could not measure the cost of discovering an appropriate relation schema from unconstrained language.

In conversation, that cost appears as paraphrase, omitted arguments, aliases, speaker references, and coreference across turns. In many such cases, a usable edge must be inferred from evidence distributed across turns rather than read directly from a single utterance. The synthetic result demonstrated correctness at a known interface. Real dialogue first required the system to infer that interface.

Reachability was never the same as ranking

The first synthetic round had already shown this distinction, and the real-text result made it harder to ignore. Edges increased the amount of reachable evidence together with the amount of irrelevant material. The conversational graphs examined here contained hubs—speakers, recurring locations, common activities, and entities discussed across months. An erroneous or broad edge can pull an entire neighborhood into the candidate set.

Reciprocal-rank fusion cannot recover information that has already been lost when noisy graph candidates displace the lexical ranking. Graph construction is not a free preprocessing step; it is a second retrieval model with its own error distribution.

Extraction errors changed the search space

The real-dialogue run surfaced speaker hubs, alias mismatches, unresolved coreference, and noisy typed facts. Filtering conversational hubs helped only marginally in the LoCoMo probe. Prefix-based entity merging could combine names that should remain separate while missing variants that referred to the same person.

These are not merely passage-scoring errors. In a flat text index, a mistaken entity interpretation changes a score. In a graph, it may create a false path and redistribute relevance across many passages. An extractor can therefore look acceptable in isolation while its errors still make the resulting graph a worse retrieval structure.

The stronger structure was hierarchical

After the graph result, I kept dense embeddings and changed where the system searched. The follow-up averaged turn embeddings into one vector per LongMemEval instance, selected the three most relevant instances, and searched individual turns only inside them.

On the same 20-instance pool, full-dimensional EmbeddingGemma scored 0.301 recall@10. Coarse-to-fine retrieval reached 0.438, a 46% relative improvement, while a term-frequency query-expansion experiment collapsed to 0.017. This does not establish coarse-to-fine retrieval as a general solution. LongMemEval itself reports that session decomposition and fact-level representations can improve multi-session reasoning. The local result supports a smaller inference: for this workload, a useful inductive bias concerned where in the conversation to search more than which entities to traverse.

How this relates to GraphRAG research

“GraphRAG” names a family of methods, not a single algorithm. The Skinki retriever is not a reproduction of Microsoft GraphRAG, HippoRAG, or LightRAG.

HippoRAG extracts an open knowledge graph and uses Personalized PageRank to retrieve on MuSiQue, 2WikiMultiHopQA, and HotpotQA. Its reported gains show that graph structure can work when the benchmark, graph construction, and traversal method align with multi-hop document QA. LightRAG combines graph and vector representations with low- and high-level retrieval. Other GraphRAG systems address global summarization rather than turn-level evidence recall.

The present experiment occupies a different regime: long, personal, temporally evolving dialogue; a small local extractor; and evidence-turn recall inside an agent-memory workload. This distinction matters because the result concerns the transfer of one retrieval design, not the viability of the wider GraphRAG family.

What can be concluded

The evidence supports a limited sequence of claims. A typed relation graph produced a large, deterministic retrieval gain on a synthetic corpus whose planted causal structure matched the retriever’s schema. That gain did not appear in the small real-dialogue evaluations performed here. Within the pooled LongMemEval subset, dense retrieval and session-level coarse-to-fine search were stronger than the tested graph variants. The practical conclusion is correspondingly narrow: synthetic multi-hop success was insufficient evidence for adopting this graph retriever in conversational memory.

Nothing in the experiment establishes that GraphRAG is ineffective, that BM25 is generally superior, that coarse-to-fine retrieval is state of the art, or that the observed differences will persist on the full benchmarks. Those stronger propositions require evidence this study did not collect.

The experiment that should come next

A publication-grade follow-up should begin by fixing its experimental contract. I would evaluate the complete LongMemEval multi-session split and the full LoCoMo QA set, storing per-query rankings, extraction artifacts, model hashes, prompts, and environment metadata in a replayable release. BM25, dense retrieval, RRF, session decomposition, coarse-to-fine retrieval, and at least one PPR-style graph baseline should then run against the same index and query set.

For that follow-up, the central quantity is a paired transfer effect rather than two isolated leaderboard numbers:

followed by the change in that effect between synthetic and real regimes. This contract would distinguish a small true benefit from noise, extraction failure, or benchmark coupling.

A failed transfer is still a result

Stopping when the synthetic score rose from 0.325 to 0.800 would have produced a cleaner story. Continuing the test produced the more useful one.

The result does not show that graphs are useless. It shows that a graph retriever is a hypothesis about the structure of relevance. Synthetic data may encode that hypothesis in its generator; real dialogue requires the system to infer the structure, tolerate extraction noise, control hubs, and still rank the right evidence above thousands of plausible neighbors.

The synthetic win was real within its test. Its failure to travel is the result worth carrying forward.