Reflections on My GraphRAG Tutorial Preparation

Hello, this is Hardy.

It’s already the end of Sunday — I hope you all had a wonderful week. Mine was exceptionally busy, with the GUG seminar and the upcoming tutorial preparation. I usually take time for a personal mid-year retrospective around this season, but had to postpone it for now. I plan to finish next week’s tutorial and then reflect on how the first half of this year has gone.

As expected, I spent this weekend preparing the tutorial again. I keep reworking the content repeatedly because I never feel fully satisfied with it. Although I’ve framed the structure, I keep worrying about whether the internal content is truly meaningful.

Anyway! The core goal of this tutorial is to examine LightRAG, a trendy GraphRAG architecture from an academic perspective, and then explore what came before it (Microsoft GraphRAG) and after it (PathRAG), including their respective features.

Interest in GraphRAG is growing across academia and industry. By reviewing which architectures are gaining attention and what distinguishes them, I hope to help those who want to pursue GraphRAG research. Broadly speaking, I see two pathways for such research:

1. Model-focused, where you restructure the architecture to improve benchmark performance, or
2. Data-focused, where you apply GraphRAG to your own domain-specific data and analyze the resulting insights.

This time, I ambitiously decided to explore both. While I may not completely overhaul the architecture, I will review prominent solutions like Microsoft GraphRAG, LightRAG, and PathRAG to catch up with trends. Then, treating data from the U.S. Trade Representative (USTR) as a hypothetical domain dataset, I will attempt entity extraction and evaluation dataset construction. That part was quite challenging to implement.

On the industry side, participants will explore skills for managing metadata and ontologies, starting with the DataHub Business Glossary — how to read/write terms, manage metadata, and link this to graph pipelines. Though not overly flashy, I strongly believe mastering the pipeline from DataHub Store to GDBMS and then to the LightRAG engine will be practically useful, so I added this as well.

Today, I’d like to share some trial-and-error lessons I encountered while building these tutorial materials. If any of the following apply to you, you may find this helpful:

1. You are curious about how to build an evaluation dataset for GraphRAG.
2. You want to try integrating a Neo4j custom knowledge graph with a LightRAG server and anticipate potential pitfalls.
3. You are wondering how ontology-based prompt engineering connects to DataHub.

Building a GraphRAG Evaluation Dataset

I started by creating a custom evaluation dataset, taking inspiration from multi-hop datasets in the GraphRAG domain. This made sense because:

1. My data was sourced from the internet.
2. The dataset assumed a multi-document context.

Among many candidates, the multi-hop dataset stood out. I’ll leave the reference link at the end of this post — it’s quite simple to understand and I highly recommend a read.

Chunk-to-chunk heatmap

First, I chunked trade-related documents published by the USTR and embedded these chunks. To measure their similarity, I constructed a 512x512 embedding matrix and used dot-product-based similarity scoring. This lets you extract the top-k most similar chunks.

To verify whether these similar chunks were meaningful, I generated a heatmap, revealing a pattern similar to a carpet. Visually checking this pattern is a very intuitive way to see if the dataset distribution is ready for a robust top-k retrieval scenario.

Why GraphRAG hierarchy works better in multi-hop tasks

When you look at a multi-hop RAG dataset, it generally defines four query types: comparison, null, temporal, and inference. These queries need supporting evidence, which is where claims come in. For example, one claim generation prompt describes a claim as

“A claim is a statement or assertion made within a text expressing a belief, opinion, or fact. Given evidence from the original context, please extract one claim and its associated topics.

This means you treat the claim as meaningful evidence within the chunk, then build queries around the claim’s topics or targets.

Does claim sound familiar? Yes — this is the same concept emphasized in Microsoft GraphRAG, where claims are extracted from text units. This consistency helps explain why hierarchical GraphRAG (local query) outperformed pure KG-based GraphRAG in systematic evaluations mentioned in the papers I reviewed and in my GUG seminar introduction. If you’re curious, I recommend comparing the two claim-generation and claim-extraction prompts — you’ll find it quite interesting.

Neo4j ↔ LightRAG Server WebUI Communication

1. According to the LightRAG GitHub documentation, it is generally better to launch the server (rather than just installing the core package) for integrations. There is a reason for this: the LightRAG server is designed with Gunicorn and exposes an API directly. Even if you don’t need the application’s web interface, you can easily reuse the API.
2. Customizing a KG ingestion format for LightRAG is very tricky. Therefore, your entity extraction prompts must be aware of and aligned with this format to produce meaningful results. In addition, if you want to leverage ontology for entity types or descriptions, you will need more prompt engineering. To sum up, it’s best to design your prompts while considering both the LightRAG custom knowledge graph ingestion format and ontology-based requirements.

Ontology-Based Prompt Engineering

1. Supply chain RDF analysis
   I parsed the IOF supply chain RDF with rdflib and reviewed its classes, individuals, properties, as well as metadata like maximum depth, number of children, and undefined classes. Network analysis tools like networkx help here. I strongly felt there should be a monitoring system to regularly track these elements and maintain RDF/metadata quality, ideally connected to DataHub.
2. Pain points with DataHub write/read
   Through the DataHub REST interface, I could only write data — not read it. As a workaround, I had to use GraphQL to search for the business terms (metadata) I needed. Although simple in hindsight, it caused a lot of frustration, so I recommend keeping this in mind if you are a first-time user.
3. Business term/glossary advantages
   The Business Term tab in DataHub lets you define relationships among terms and map which domain team uses them, through a friendly GUI. This means that even domain experts with no coding skills can easily document their knowledge. As this implicit knowledge is accumulated and updated, you can transform it into a knowledge graph and then apply it to GraphRAG, making this a fantastic pipeline.
4. Prompt management challenge
   I ended up with eight separate prompts:

• Query generation: claim, comparison, inference, null, temporal
• Entity extraction: knowledge graph building
• Evaluation: LLM-based judgments

-> Managing these without a platform feels chaotic. You really need a prompt-sharing platform — I personally think Opik is the best.

1. API cost experiments
   I worried a lot about API costs, but after finally testing, I saw it cost me only about $20 to build my demo customized dataset. My advice? Just do it! Once you see token usage distribution and token input/output ratios, you can better estimate future costs.
2. Entity extraction output inconsistency
   Even with a clear prompt, the entity extraction outputs were inconsistent. To handle this, you need a robustness parser to supplement these variations, or explicitly define a strict structure and tone in the prompt to maintain consistent results. (In hindsight, I really missed using the LangChain output parser for this.)

I hope this lesson-learned will help you who wants to integrate or use the ‘LightRAG’ framework. if you wonder about this post and experience feel free to comment or message me.

Linkedin : linkedin.com/in/yitaejeong

# Reference

1.Multihop RAG dataset — MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries , https://arxiv.org/pdf/2401.15391
2.Microsoft claim extraction prompt — https://github.com/microsoft/graphrag/blob/main/graphrag/prompts/index/extract_claims.py
3.Supplychain IOF — https://github.com/iofoundry/ontology/tree/master/supplychain