Public Case: LinkedIn Knowledge-Graph RAG for Support

Public case (sourced). Based on LinkedIn authors' published paper, "Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering" (Xu et al., SIGIR '24, arXiv 2404.17723); every fact below is attributed to it.

This case belongs to Data and Retrieval Architecture for AI Systems. Where the paper is silent, this page says "the paper does not say" and does not fill the gap. Component names in the diagrams are this course's labels; the paper does not name its services.

Context

The paper comes from seven authors at LinkedIn. It addresses customer-service technical support, where, in the authors' words, "swiftly and accurately retrieving relevant past issues is critical for efficiently resolving customer inquiries". The knowledge source is a body of historical issue-tracking tickets. The authors name Jira as an example of such a system. The paper is five pages long.

Problem statement

The authors state two limitations of the conventional approach to retrieval-augmented generation (RAG) over tickets:

1. Compromised retrieval accuracy from ignoring structure. Issue tickets have inherent structure and links to each other ("issue A is related to/copied from/caused by issue B"). Compressing them into text chunks, the authors say, loses that information.
2. Reduced answer quality from segmentation. Fixed-length segments can disconnect related content. Their example is a ticket that describes the issue at the start and the solution at the end, which may be split so that critical parts of the solution are missing.

The paper's goal is a retrieval method that keeps intra-issue structure and inter-issue relations, and it measures success with retrieval metrics (MRR, Recall@K, NDCG@K), text-generation metrics (BLEU, ROUGE, METEOR) and a production measure, per-issue resolution time.

Constraints

The paper states these constraints:

• Embedding models have context-length limits, which is why conventional pipelines segment tickets.
• The system must serve online: the authors describe a fallback to text-based retrieval "for robust online serving" if a query fails to execute.

The paper does not say anything about latency targets, cost, data privacy, access control or ticket volume. It does not say which vector or graph database LinkedIn runs: Qdrant and Neo4j appear only as examples ("for instance", "such as").

Why the traditional approach fell short

The baseline in the paper is conventional text-based embedding-based retrieval (EBR). The authors describe it as two stages: tickets are treated as plain text, split into chunks that fit the embedding model, and each chunk becomes an embedding vector; at question time, the most relevant chunks are retrieved and fed to an LLM as context. The authors attribute its weaknesses to the two limitations above: lost structure and links, and segmentation that separates a problem from its solution. In the experiments the baseline and the new method used the same LLM (GPT-4) and the same embedding model (E5).

Options considered

The paper does not present a formal options analysis. The alternatives it evaluates or mentions are:

The related-work section also names three families of question answering over knowledge graphs; the paper does not evaluate them. For a reader, the manual process is the non-AI option and the text baseline is the simpler AI option.

Solution architecture

The authors describe two phases: constructing the knowledge graph from historical tickets, and retrieval and answering at query time.

Figure 1. Knowledge graph construction and the query path, as described in the paper

How to read this: the left side is built ahead of time from tickets, the right side runs per question, and the dotted arrow is the fallback the paper describes.

Figure 2. The runtime path in the paper's own order

How to read this: the loop is the entity-by-entity matching, the note is the ticket ranking, and the alt block is the documented fallback.

How AI solved it, step by step

This follows the paper's description (its Section 3 and Figure 1). The example is the paper's own.

1. Build the tree. Each historical ticket becomes a tree whose nodes are ticket sections such as Summary, Description and Priority. The authors say they use a hybrid method: rule-based extraction for predefined fields such as code sections found by keywords, and an LLM, guided by a YAML template of the sections support staff routinely use, for text the rules cannot parse.
2. Link tickets. Explicit links come from fields defined in the tracker. Implicit links connect tickets whose title embeddings pass a cosine-similarity threshold. In the paper's Figure 1, ticket ENT-22970 has a direct clone link to PORT-133061 and is implicitly connected with ENT-1744 and ENT-3547.
3. Embed nodes. Text-rich nodes (summary, description, steps to reproduce) are embedded with models such as BERT and E5 and stored in a vector database; the paper gives Qdrant as an example.
4. Parse the question. An LLM turns the query into entities (template sections mapped to extracted values) and intents (the sections the answer should cover). Decision: which parts of the question describe the issue and which say what the user wants.
5. Rank tickets. For each entity, cosine similarity is computed between its value and the nodes of the matching section. Node scores are summed per ticket and the top tickets are kept.
6. Query the graph. The LLM rewrites the question with the ticket id and translates it into a graph query language such as Cypher for Neo4j. Decision: which relations to walk from that ticket.
7. Answer. The LLM writes the answer from the retrieved values and the original question. If query execution has problems, the system reverts to text-based retrieval.

Under the hood

The central step is ranking tickets by section-specific embedding matches rather than by whole-chunk similarity. The paper says that for each entity pair (a section key and a value) cosine similarity is computed between the value and all graph nodes of that section, and node scores are summed per ticket to get a ticket-level score. Its stated premise is that "the occurrence of multiple query entities is indicative of pertinent links". This page's reading is that a ticket matching both a summary entity and a description entity collects two terms in the sum, so it tends to score above a ticket matching only one. The paper does not report the number of tickets kept, or the similarity threshold for implicit links.

The second step is retrieval by graph query. For the paper's example, "how to reproduce 'ENT-22970'", the generated Cypher is:

MATCH (j:Ticket {ticket_ID: 'ENT-22970'})
-[:HAS_DESCRIPTION]-> (description:Description)
-[:HAS_STEPS_TO_REPRODUCE]-> (steps_to_reproduce: StepsToReproduce)
RETURN steps_to_reproduce.value

Because the answer is read from a named section of one ticket, a problem description and its fix are not separated by a chunk boundary. The authors say that preserving the logical coherence of ticket sections is how their method overcomes the second limitation. The cosine-similarity mechanism behind step 5 is explained in Under the hood: Embeddings and similarity metrics.

What went wrong / failure modes

What the paper reports. It names one failure path: if query execution encounters issues, a fallback reverts to baseline text-based retrieval. It lists future work: an automated mechanism for extracting graph templates (the current template reflects sections support staff routinely use), dynamic updates of the graph based on user queries, and testing the approach beyond customer service.

What the paper does not say. It does not report how often the fallback fires, how often the generated graph query is wrong, how wrong answers were handled, the size of the evaluation set, the number of tickets kept, the implicit-link threshold, how the graph is refreshed as new tickets arrive, or how access to tickets is controlled.

Failure modes a team adopting this should plan for (this page's reading, not claims from the paper). The paper's design implies several dependencies: an LLM-generated query that can fail or return the wrong section, a fixed template of ticket sections that must match how tickets are written, and a graph that is only as current as its last build. Each deserves a test and a named fallback before the pattern is copied.

Results

Only numbers published in the paper are given. The offline evaluation used a curated "golden" dataset of typical queries, support tickets and authoritative solutions; the paper does not give its size. In that evaluation both the baseline and the new method used GPT-4 and the E5 embedding model.

Retrieval (the paper's Table 1)

Answer quality (Table 2)

The authors report that the method surpasses the baseline by 77.6% in MRR and by 0.32 in BLEU. The check against the table is (0.927 - 0.522) / 0.522 = 0.776 and 0.377 - 0.057 = 0.32.

Deployment (Table 3). The authors say the method was deployed in LinkedIn's customer service team for approximately six months, across multiple product lines. The team was split randomly into one group that used the system and one that used traditional manual methods.

The authors report a 28.6% reduction in median resolution time per issue, which matches (7 - 5) / 7 = 0.286. The paper does not report group sizes, ticket counts or a significance test.

Tradeoffs and when NOT to use this approach

The paper has no limitations section. Its future-work paragraph names three open items: an automated mechanism for extracting graph templates, dynamic updates of the graph based on user queries, and applicability beyond customer service. This page reads these as things the current system does not yet do. The paper describes its evaluation data only as a curated golden dataset and does not say how it was built or how large it is.

This page's reading: the method assumes source documents with stable, named sections and explicit links. If the corpus is free-form prose without that structure, there is nothing to parse into a tree, and a plain hybrid retrieval pipeline is the simpler choice. If tickets change constantly, plan how and how often the graph is rebuilt. The paper gives no cost or latency figures, so compare those on your own data.

Design rules extracted

These rules are drawn from the paper's design, not stated by the authors as rules.

1. Parse documents along the structure they already have before chunking, so a problem and its solution stay together.
2. Keep explicit links between documents as graph edges, and add similarity links only with a threshold you tune.
3. Match each part of the question to the field it describes, not to whole-document text.
4. Keep a fallback to simpler retrieval for when a generated graph query fails.
5. Measure retrieval and generation with separate metrics offline, then confirm the effect with a randomised comparison in production.

Sources

• Zhentao Xu, Mark Jerome Cruz, Matthew Guevara, Tie Wang, Manasi Deshpande, Xiaofeng Wang and Zheng Li, "Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering", SIGIR '24, July 14-18, 2024, Washington, DC. Full text read in the arXiv v2 PDF (6 May 2024): https://arxiv.org/pdf/2404.17723
• Abstract page: https://arxiv.org/abs/2404.17723
• ACM record named in the paper: https://doi.org/10.1145/3626772.3661370

Last verified: 2026-10-22