THE ACHILLES’ HEEL OF RAG: IT CAN FIND INFORMATION, BUT IT MAY NOT AGGREGATE IT CORRECTLY
QUICK SUMMARY
RAG works well when the answer can be found in a few specific passages.
However, when users want to identify trends, count cases, or aggregate information across thousands of documents, the system must do much more than retrieve relevant text.
GraphRAG helps AI identify entities and understand the relationships between them. But knowing “which pieces of information are connected” is not the same as accurately determining “how many there are, what the total is, or what percentage they represent.”
In particular, when the source documents do not already contain aggregated reports, aggregation becomes a large-scale data processing problem rather than a simple retrieval task.
HOW MUCH DATA DOES RAG ACTUALLY SEE?
A typical RAG system follows a process like this:
Documents
- Split the documents into smaller chunks.
- Retrieve the most relevant chunks.
- Send those chunks to an AI model.
- Generate an answer.
This approach works well for specific questions such as:
“How long does the refund policy remain valid?”
The answer may exist in a single passage. The system only needs to retrieve the correct one.
Now consider a different question:
“What were the three most common customer complaints over the past year?”
To answer accurately, the AI must examine enough data, group similar responses, remove duplicates, and compare the frequency of each category.
This is no longer simply a matter of finding one relevant passage.
The paper From Local to Global: A Graph RAG Approach to Query-Focused Summarization describes such requests as global questions—questions that require an understanding of the broader picture across an entire document collection.
The authors argue that traditional RAG struggles with these questions because they are closer to query-focused summarization than to retrieving a small number of relevant passages.
Source:
https://arxiv.org/abs/2404.16130
GRAPHRAG CONNECTS THE PIECES
One limitation of vector search is that document chunks are often treated as relatively independent units.
In reality, information often exists as a chain of relationships:
Customer A
purchased Product X
experienced a payment issue
contacted customer support
requested a refund
These facts may be spread across multiple emails, support tickets, and reports.
GraphRAG addresses this problem by:
- Extracting entities and relationships from documents.
- Building a knowledge graph.
- Grouping related entities into content communities.
- Generating a summary for each community.
This structure provides better support for questions that require a global view of the data.
In the experiments presented in From Local to Global, GraphRAG produced more comprehensive and diverse answers than baseline RAG for a set of qualitative, global-level questions.
In simple terms, GraphRAG can help AI understand:
- Who is connected to whom?
- Which events are associated with which products?
- Which topics frequently appear together?
- How are different issues across the document collection connected?
This is a significant improvement over retrieving passages based only on semantic similarity.
UNDERSTANDING RELATIONSHIPS DOES NOT GUARANTEE ACCURATE CALCULATION
Suppose a company has 20,000 support tickets and asks:
“How many customers experienced payment issues during the last quarter?”
To answer correctly, the system must:
- Identify all relevant tickets.
- Distinguish payment issues from login or refund issues.
- Recognize when multiple tickets refer to the same incident.
- Remove duplicate records.
- Apply the correct date range.
- Count the results across the entire dataset.
GraphRAG can connect customers, transactions, products, and incidents.
However, the original GraphRAG architecture produces global answers by combining natural-language community summaries.
The paper was not designed to demonstrate that GraphRAG can reliably perform operations such as:
- COUNT
- SUM
- AVG
- Percentage calculations
- Time-based numerical comparisons
across an entire corpus.
It would therefore be incorrect to assume:
“Because GraphRAG performs well on global questions, it will also calculate every number across the document collection correctly.”
Summarizing a trend and calculating an exact number are two different problems.
The paper Structured RAG for Answering Aggregative Questions argues that most RAG systems are optimized for cases in which only a small portion of the corpus contains the necessary information.
They struggle when the system must collect facts from many documents and then perform aggregation across them.
The proposed S-RAG method creates a structured representation during the ingestion stage and translates natural-language questions into formal queries over that structure.
Source:
https://arxiv.org/abs/2511.08505
WHEN THE DOCUMENTS DO NOT CONTAIN PRE-AGGREGATED INFORMATION
RAG has a much easier task when the documents already contain a summary such as:
“In the third quarter, the company received 1,240 support requests, 28% of which were related to payments.”
The system only needs to retrieve the correct passage.
In practice, however, the data often consists only of individual records:
Ticket 1: Unable to pay by card
Ticket 2: Payment was deducted, but the order was not confirmed
Ticket 3: The account was charged twice
…
Ticket 20,000
No document directly states the total number of payment issues.
To answer the question, the system must build the aggregation itself:
- Collect records
- Extract facts
- Normalize different expressions
- Classify the records
- Remove duplicates
- Perform calculations
- Validate the result
This is a significant challenge because retrieval must achieve sufficiently high recall.
If even part of the relevant dataset is missed, the resulting count or percentage may already be incorrect.
The benchmark introduced in From Chat Logs to Collective Insights: Aggregative Question Answering studies aggregation questions across more than 182,000 real-world conversations.
The results show that current methods typically face one of two problems:
- They struggle to reason over data at scale.
- They require very high computational costs to process enough information.
This suggests that aggregation over unstructured data remains an open problem.
Source:
https://arxiv.org/abs/2505.23765
CAN WE SOLVE THE PROBLEM BY RETRIEVING MORE CHUNKS?
A simple approach is to increase the number of retrieved passages, for example from 5 chunks to 100.
However, retrieving more information does not necessarily produce better results.
The system may encounter new problems:
- More duplicate information.
- Longer and noisier context.
- Higher processing costs.
- Greater difficulty identifying which facts should be calculated.
- Relevant but less semantically similar groups may still be missed.
The paper RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval addresses part of this problem by organizing and summarizing documents at multiple levels.
The system can retrieve both detailed passages and higher-level summaries rather than searching through a flat list of chunks.
However, RAPTOR primarily improves contextual understanding and retrieval. It does not replace a deterministic calculation tool.
Source:
https://arxiv.org/abs/2401.18059
WHAT SHOULD A PRACTICAL ARCHITECTURE LOOK LIKE?
A reliable system should first identify what kind of question the user is asking.
When the user wants to retrieve information
Example:
“What are the conditions for receiving a refund?”
Traditional RAG combined with reranking may be sufficient.
When the user wants to understand relationships
Example:
“Which products are commonly associated with payment issues?”
GraphRAG may be more suitable because it organizes entities and the relationships between them.
When the user wants numerical aggregation
Example:
“What was the total value of failed transactions this quarter?”
The system should allow AI to interpret the request and create a plan, while delegating filtering, counting, and calculation to deterministic tools such as:
- SQL or analytical databases.
- Dataframes.
- Search engines with aggregation capabilities.
- Knowledge graphs with clearly defined schemas.
- Controlled code execution environments.
A practical pipeline may look like this:
- User submits a question
- AI classifies the question type
- RAG or GraphRAG gathers supporting evidence
- A structured tool filters and calculates the data
- The system validates the result
- AI explains the result in natural language
CONCLUSION
- RAG is strong at retrieving information.
- GraphRAG goes one step further by helping systems understand relationships and form a qualitative overview of the data.
- However, retrieval and summarization are still insufficient for questions that require exact numbers, such as counts, totals, averages, or percentages across an entire document collection.
- When the source data does not already contain aggregated reports, the system must transform thousands of unstructured documents into information that can be calculated.
- At that point, the problem is no longer just RAG. It becomes a complete data processing and analytics pipeline.
Before choosing an architecture, organizations should determine whether users are trying to:
“Find a piece of information, understand a relationship, or calculate something across the entire dataset?”
These questions may look similar in a conversational interface, but they require very different systems behind the scenes.
REFERENCES
Darren Edge et al.
From Local to Global: A Graph RAG Approach to Query-Focused Summarization.
arXiv:2404.16130.
https://arxiv.org/abs/2404.16130
Parth Sarthi et al.
RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval.
arXiv:2401.18059.
https://arxiv.org/abs/2401.18059
Omri Koshorek et al.
Structured RAG for Answering Aggregative Questions.
arXiv:2511.08505.
https://arxiv.org/abs/2511.08505
Wentao Zhang, Woojeong Kim, and Yuntian Deng.
From Chat Logs to Collective Insights: Aggregative Question Answering.
arXiv:2505.23765.
https://arxiv.org/abs/2505.23765