From Chunks to Graphs: An Overview of RAG Architectures

Retrieval-Augmented Generation has become the default architecture for grounding LLM responses in external knowledge. According to Forrester's 2025 analysis, RAG is now the standard for enterprise knowledge assistants. But RAG is not monolithic - it has evolved significantly.

In this post, I'll walk through three distinct approaches: Traditional RAG, GraphRAG, and LightRAG. We'll look at how each works, when to use them, and the real trade-offs you'll face in production.

1. The Problem Space

Large language models have fundamental limitations: hallucinations, knowledge cutoffs, and finite context windows. RAG addresses these by retrieving relevant information at query time and injecting it into the prompt.

The Evolution

RAG has gone through several generations:

1. Naive RAG (2020-2022): Simple chunk-and-retrieve
2. Advanced RAG (2022-2023): Better chunking, reranking, query rewriting
3. Modular RAG (2023-2024): Composable pipelines, routing
4. Graph-based RAG (2024-present): Knowledge graphs meet retrieval

The Challenge

Balancing four competing concerns:

• Accuracy: Does the system retrieve the right information?
• Cost: How many tokens and API calls per query?
• Latency: How fast is the response?
• Maintainability: How easily can the knowledge base be updated?

Different RAG architectures make different trade-offs across these dimensions.

2. Traditional RAG

The original RAG architecture follows a linear pipeline.

Processing Pipeline

1. Document Loading: Ingest raw documents (PDF, HTML, TXT)
2. Chunking: Split into segments (100-500 tokens)
   • Fixed-size: Split at token count boundaries
   • Recursive: Split by separators (paragraphs → sentences → words)
   • Semantic: Split at topic boundaries using embeddings
   • Sentence-based: Preserve complete sentences
3. Embedding: Convert chunks to dense vectors (OpenAI, Cohere, local models)
4. Indexing: Store in vector database (FAISS, Pinecone, Weaviate, Chroma)
5. Retrieval: Query embedding → cosine similarity → top-k chunks
6. Augmentation: Inject retrieved context into prompt
7. Generation: LLM produces response from retrieved context

Strengths

• Simple to implement and debug
• Low latency (~120ms)
• Mature ecosystem with extensive tooling
• Cost-effective for simple queries

Weaknesses

Limitation	Impact
Information Loss	Arbitrary chunking disrupts semantic boundaries
Flat Representation	No relationship modeling between concepts
Context Fragmentation	Related information scattered across chunks
Single-Hop Only	Cannot traverse relationships for complex queries
No Global Understanding	Cannot answer thematic questions
Redundancy	Same information duplicated across overlapping chunks

When Traditional RAG Fails

Traditional RAG breaks down on three types of queries:

• Multi-hop reasoning: "How does X relate to Y through Z?"
• Thematic questions: "What are the key trends in this dataset?"
• Cross-document synthesis: "Compare perspectives across all sources"

Ask "What are the main themes in this dataset?" and traditional RAG has no good answer - it can only return the most similar chunks, not synthesize across all of them.

3. Solution 1: GraphRAG

Microsoft Research released GraphRAG in 2024 to address these limitations. The core insight: knowledge graphs preserve relational structure that chunking destroys.

Processing Pipeline

Stage 1: Chunking Document segmentation, similar to traditional RAG but with smaller chunks to improve entity extraction accuracy.

Stage 2: Entity & Relationship Extraction LLM extracts entities (nodes) and relationships (edges) from each chunk. Output: knowledge graph triples like (Marie Curie) → [discovered] → (Radium).

Stage 3: Community Detection This is the key differentiator. The Leiden algorithm clusters related entities into hierarchical communities. These groupings enable thematic understanding across the entire dataset.

Stage 4: Community Summarization LLM generates summary reports for each community at multiple levels, from fine-grained to high-level themes.

Stage 5: Query Processing Two distinct modes based on query type.

Query Modes

Global Search handles holistic questions about the entire dataset:

1. Query all community summaries in parallel
2. Map-reduce aggregation across communities
3. Synthesize global answer from partial responses

Example: "What are the main themes in this dataset?"

Cost: High (~610K tokens, hundreds of API calls)

Local Search handles specific questions about entities:

1. Find relevant entities from query
2. Fan out to neighboring nodes (multi-hop)
3. Gather local context from subgraph
4. Generate answer from local information

Example: "What are Scrooge's main relationships?"

Cost: Lower than global, but still significant

Trade-offs

Strengths:

• Deep relational understanding
• Excellent global/thematic queries
• Multi-hop reasoning capability
• Automatic pattern and community discovery
• Microsoft enterprise backing

Limitations:

• Extremely high query cost (~610K tokens)
• Hundreds of API calls per query (rate limit risk)
• Slow indexing due to community detection
• Full rebuild required for updates
• Complex infrastructure

Technical Deep Dive

Graph Database Options:

• Neo4j: Production-grade, ACID compliant, Cypher query language
• NetworkX: Python library, good for prototyping, in-memory only
• Custom: JSON/pickle serialization for simpler deployments

Community Detection Algorithms:

• Leiden: Default choice, hierarchical clustering, better modularity than Louvain
• Louvain: Faster but less accurate, good for initial experiments

Embedding Integration: GraphRAG can operate in hybrid mode, combining graph traversal with vector similarity for entity matching. This improves recall when entity names vary across documents.

Hierarchical Summary Strategy: Communities are summarized at multiple levels (e.g., 3-5 levels). Higher levels capture broader themes, lower levels preserve detail. Query routing determines which level to access based on question scope.

4. Solution 2: LightRAG

In October 2024, researchers from Hong Kong University released LightRAG specifically to address GraphRAG's cost and update limitations.

Core Approach

LightRAG combines knowledge graphs with vector retrieval using dual-level key-value pairs. This enables fast, cost-effective retrieval without expensive community clustering.

Key Features

1. Dual-level retrieval (Low + High)
2. Vector-based search at query time
3. Incremental updates (append-only)
4. 6000x fewer tokens per query

Processing Pipeline

Stage 1: Entity & Relationship Extraction LLM call per chunk, similar to GraphRAG. Extracts entities and relationships.

Stage 2: Dual-Level Key-Value Indexing LLM generates keys for each entity/relationship:

• Low-level keys: Specific entity identifiers
• High-level keys: Thematic/conceptual descriptors

Stage 3: Vector Embedding Entity and relationship descriptions are embedded and stored in a lightweight vector database.

Stage 4: Query Processing Vector search retrieves relevant entities/relationships. No LLM needed for retrieval. Single LLM call for final answer generation.

Dual-Level Retrieval

Low-Level Retrieval (Precision)

Aspect	Description
Scope	1-hop direct connections
Matching	Exact keyword matching
Output	Factual, precise results
Use Case	"Who is the CEO of Tesla?"

High-Level Retrieval (Context)

Aspect	Description
Scope	2-3 hop neighbor expansion
Matching	Conceptual/semantic matching
Output	Comprehensive context
Use Case	"How does EV industry affect climate?"

Hybrid Mode combines both for balanced precision and context.

Trade-offs

Strengths:

• 90%+ cost reduction at query time
• 30% faster query response than traditional RAG
• Incremental updates without full rebuild
• Simple architecture, easier to debug
• Fully open source and customizable

Limitations:

• Newer technology (October 2024), less battle-tested
• Smaller ecosystem and community
• No automatic community detection
• May miss complex multi-hop relationships that require global context

Technical Deep Dive

Three Storage Backends:

LightRAG uses a pluggable storage architecture with three distinct backends:

Storage Type	Purpose	Production Options
KV Storage	Entity/relation metadata	Redis, PostgreSQL, MongoDB, JSON
Vector Storage	Embedding similarity search	Milvus, Qdrant, pgvector, nano-vectordb
Graph Storage	Relationship traversal	Neo4j, PostgreSQL, NetworkX

For production at scale (1M+ records), the recommended stack is Redis + Milvus + Neo4j. The default JSON/nano-vectordb/NetworkX setup is only suitable for development.

Entity Extraction (from codebase):

The extraction prompt uses a structured format with delimiters:

entity<|#|>entity_name<|#|>entity_type<|#|>description
relation<|#|>source<|#|>target<|#|>keywords<|#|>description

Entity types are configurable (default: Person, Organization, Location, Event, Concept, Method, etc.). Each chunk goes through extraction + optional "gleaning" pass to catch missed entities.

Query Context Building (4-stage pipeline):

1. Search: Vector similarity on entities/relations using extracted keywords
2. Truncate: Apply token limits (default: 6K entity, 8K relation, 30K total)
3. Merge: Combine chunks from matched entities, deduplicate
4. Build: Format final LLM context with references

Token Control System:

MAX_ENTITY_TOKENS=6000   # Entity context budget
MAX_RELATION_TOKENS=8000 # Relation context budget
MAX_TOTAL_TOKENS=30000   # Total including chunks
TOP_K=40                 # Entities/relations retrieved

Keyword Extraction at Query Time:

# From operate.py - keywords drive retrieval routing
hl_keywords, ll_keywords = await get_keywords_from_query(query, ...)

# Local mode: uses ll_keywords (specific entities)
# Global mode: uses hl_keywords (concepts/themes)
# Hybrid mode: combines both

Caching Strategy:

LightRAG caches both extraction results and query responses:

• llm_response_cache: Stores extraction results per chunk
• Query cache: Hashes query + params to avoid duplicate LLM calls

Incremental Updates:

New documents process independently with entity deduplication:

• Entities matched by name (case-insensitive)
• Descriptions merged when entity already exists
• Source IDs tracked per entity (FIFO limit: 300 chunks)
• No graph rebuild required

Why LightRAG Still Has Graph Storage

If LightRAG doesn't use community detection, why does it need a graph at all?

Different Purpose: Local Traversal vs Global Analysis

Aspect	GraphRAG Graph	LightRAG Graph
Purpose	Global structure analysis	Local traversal from entry points
Entry Point	Community summaries	Vector-matched entities
Traversal	Entire community hierarchy	1-hop from matched nodes
Community Detection	Required (Leiden)	Not used

The Retrieval Flow:

1. Vector DB (primary)     → Find candidate entities by embedding
                              ↓
2. Graph Storage           → Get entity metadata (descriptions)
                           → Get node degrees (rank by connectivity)
                           → Fan out to connected edges (1-hop)
                           → Get relationship data (keywords, descriptions)

What the graph provides:

• Entity metadata: Full descriptions stored on nodes
• Connectivity ranking: Node degree = importance (more connected = more relevant)
• Relationship expansion: From matched entities, find all connected relationships
• Edge data: Relationship descriptions and keywords for context

Why no community detection needed:

• LightRAG doesn't answer "what are ALL the themes in this corpus?"
• It starts from specific vector-matched entities, not a global view
• Local 1-hop expansion captures enough context for most queries
• Avoids the O(n) rebuild cost of maintaining community structure

Important Clarification

Indexing still requires LLM calls, similar to GraphRAG. The 6000x savings is entirely in the query phase, not indexing. This distinction matters because query costs compound with every user interaction.

5. Why GraphRAG is Expensive (And How LightRAG Fixes It)

Understanding the architectural difference is key to choosing the right approach.

GraphRAG's Two Problems

Problem 1: Full Reconstruction on Updates

GraphRAG uses the Leiden algorithm to cluster entities into hierarchical communities. Each community gets an LLM-generated summary. When new documents arrive:

1. New entities may shift cluster boundaries
2. Community memberships change
3. All affected summaries become stale
4. Must regenerate summaries for changed communities

This creates O(n) rebuild cost - adding one document can trigger reprocessing of the entire graph.

Problem 2: High Query Cost

For global queries ("What are the main themes?"), GraphRAG must:

1. Query ALL community summaries in parallel
2. Each community = one LLM call
3. Map-reduce to aggregate partial answers
4. Hundreds of communities = hundreds of API calls

This is why global search costs ~610K tokens per query.

LightRAG's Key Optimizations

Problem	GraphRAG Approach	LightRAG Solution
Structure	Community detection (Leiden)	No clustering - direct indexing
Summaries	LLM summary per community	No community summaries
Updates	Rebuild affected communities	Append-only, merge by entity name
Retrieval	Query all communities (LLM)	Vector similarity (no LLM)
Answer	Map-reduce across communities	Single LLM call

The Query Flow Comparison:

GraphRAG:  Query → [LLM × N communities] → Aggregate → Answer
LightRAG:  Query → [LLM keywords] → Vector search → [LLM generate]

GraphRAG's community structure enables deep thematic understanding but creates structural dependencies. LightRAG trades community discovery for:

• O(1) updates instead of O(n) rebuilds
• 2 LLM calls per query instead of hundreds
• ~100 tokens retrieval cost instead of ~610K

The trade-off: LightRAG cannot answer "what patterns exist across this entire corpus?" as effectively, but handles 99% of queries at 0.01% of the cost.

6. Comparative Analysis

LLM Usage Comparison

Phase	GraphRAG	LightRAG
Extract entities/relations	LLM per chunk	LLM per chunk
Index generation	LLM per community	Embedding only
Query (retrieval)	~610,000 tokens	~100 tokens
API calls per query	Hundreds	2 (keywords + answer)

At 1000 queries/day, that's $600 vs $0.10 in token costs.

Performance Metrics

Metric	Traditional RAG	GraphRAG	LightRAG
Query Latency	~120ms	2x baseline	~80ms (30% faster)
Query Token Cost	Low (~1K)	Very High (~610K)	Low (~100)
Indexing Cost	Low	High	High
Incremental Updates	Fast	Full rebuild	Append only
Setup Complexity	Simple	Complex	Moderate

Capability Matrix

Capability	Traditional	GraphRAG	LightRAG
Direct fact lookup	Excellent	Good	Good
Multi-hop reasoning	Poor	Excellent	Good
Global/thematic queries	Poor	Excellent	Good
Entity relationships	Poor	Excellent	Good
Community discovery	None	Excellent	None
Real-time updates	Excellent	Poor	Excellent
Cost efficiency	Excellent	Poor	Excellent

7. Decision Framework

When to Choose GraphRAG

Choose when:

• Budget flexibility allows higher per-query costs
• Enterprise requirements need Microsoft backing
• Knowledge base is relatively static
• Users ask global/thematic questions frequently
• Pattern and community discovery is valuable
• Complex multi-hop reasoning is critical

Avoid when:

• Cost per query is a hard constraint
• Data updates frequently (daily/weekly)
• Low latency (<100ms) is required
• Infrastructure simplicity is preferred

When to Choose LightRAG

Choose when:

• Cost sensitivity at scale
• Startup/MVP phase requiring quick deployment
• Dynamic, frequently updated knowledge base
• Speed and user experience are priorities
• Processing 100K+ documents
• Experimenting with graph RAG concepts

Avoid when:

• Community/cluster discovery is essential
• Maximum relational depth required
• Need extensive enterprise support
• Very complex multi-hop reasoning across entire corpus

When to Choose Traditional RAG

Choose when:

• Simple fact lookup queries dominate
• Minimal infrastructure desired
• Small document corpus (<1K docs)
• Rapid prototyping needed
• No relational queries expected

Avoid when:

• Users ask "why" or "how" questions
• Information spans multiple documents
• Thematic or summary queries are common

8. Implementation Considerations

Infrastructure Requirements

Component	Traditional	GraphRAG	LightRAG
Vector DB	Required	Optional	Required
Graph DB	None	Recommended	Optional
LLM API	Generation only	Heavy usage	Moderate
Compute	Low	High	Moderate

Integration Patterns

Standalone Deployment: Single RAG system handles all queries. Simplest to implement and maintain.

Hybrid Approach (Traditional + Graph): Route simple queries to traditional RAG, complex queries to graph-based. Use query classification to determine routing. Balances cost and capability.

Agentic Orchestration: RAG as a tool within an agent framework. Agent decides when to retrieve, which RAG to use, and how to combine results. Most flexible but highest complexity.

Evaluation Metrics

When benchmarking RAG systems, measure:

• Faithfulness: Does the answer accurately reflect retrieved context?
• Answer Relevance: Does the response address the query?
• Context Relevance: Is the retrieved context appropriate?
• Latency: Time to first token and total response time
• Cost: Tokens consumed per query

9. Future Directions

Emerging Approaches (2025)

• GFM-RAG: Graph Foundation Model integration
• KET-RAG: Knowledge-Enhanced Traversal
• NodeRAG: Node-centric retrieval optimization
• Agentic RAG: Multi-agent orchestration with RAG

Open Research Questions

• Optimal graph construction strategies
• Balancing indexing vs query costs
• Hybrid retrieval mechanisms
• Automated architecture selection based on query patterns

Industry Trends

What I'm seeing in production deployments:

• RAG as default: Most enterprise AI projects now start with RAG, not fine-tuning
• Graph-aware retrieval going mainstream: Even traditional RAG systems are adding relationship awareness
• Cost optimization driving adoption: LightRAG's approach resonates because query costs matter at scale
• Hybrid architectures emerging: Companies running multiple RAG types with intelligent routing

10. Conclusion

RAG is not one thing. Traditional RAG offers simplicity and speed. GraphRAG provides deep relational understanding at high cost. LightRAG balances graph-based reasoning with practical economics.

The right choice depends on your constraints:

Scenario	Recommendation
Startup MVP	LightRAG
Enterprise static KB	GraphRAG
Simple Q&A bot	Traditional RAG
Cost-sensitive scale	LightRAG
Research/Discovery	GraphRAG
Frequent updates	LightRAG

Choose based on your query patterns, budget constraints, and update frequency - not hype.

---

The field is moving fast. GraphRAG established that graphs matter for RAG. LightRAG proved you don't need to pay GraphRAG prices to get graph benefits. The next iteration will likely push both dimensions further.

entity<|#|>entity_name<|#|>entity_type<|#|>description
relation<|#|>source<|#|>target<|#|>keywords<|#|>description

MAX_ENTITY_TOKENS=6000   # 实体上下文预算
MAX_RELATION_TOKENS=8000 # 关系上下文预算
MAX_TOTAL_TOKENS=30000   # 包含块的总预算
TOP_K=40                 # 检索的实体/关系数

# 来自 operate.py - 关键词驱动检索路由
hl_keywords, ll_keywords = await get_keywords_from_query(query, ...)

# Local模式：使用 ll_keywords（具体实体）
# Global模式：使用 hl_keywords（概念/主题）
# Hybrid模式：两者结合

1. 向量数据库（主要）   → 通过嵌入找到候选实体
                           ↓
2. 图存储              → 获取实体元数据（描述）
                       → 获取节点度数（按连接度排序）
                       → 扩展到连接的边（1跳）
                       → 获取关系数据（关键词、描述）

GraphRAG:  查询 → [LLM × N个社区] → 聚合 → 答案
LightRAG:  查询 → [LLM提取关键词] → 向量搜索 → [LLM生成]