Retrieval-Augmented Generation, commonly called RAG, allows AI applications to answer questions using external data such as:
- PDFs
- Company documents
- Help-center articles
- Product documentation
- Research papers
- Knowledge bases
- Website content
But there is an important problem.
Documents are often too large to search and send directly to a language model.
A 100-page PDF may contain thousands of words and many unrelated topics.
Instead of treating the entire document as one piece, RAG systems usually divide it into smaller sections.
This process is called chunking.
Chunking is one of the most important parts of a RAG pipeline because poor chunking can lead to poor retrieval, even when you use a good embedding model and a powerful LLM.
What Is Chunking in RAG?
Chunking is the process of splitting large documents into smaller pieces of text before creating embeddings and storing them for retrieval.
Each smaller piece is called a chunk.
For example, imagine a long document:
Employee Handbook
Section 1: Working Hours
Section 2: Leave Policy
Section 3: Remote Work
Section 4: Expenses
Section 5: Termination Policy
Instead of storing the entire handbook as one searchable unit, a RAG system may split it into:
Chunk 1:
Working Hours
Chunk 2:
Leave Policy
Chunk 3:
Remote Work
Chunk 4:
Expenses
Chunk 5:
Termination Policy
When a user asks:
How many annual leave days do employees receive?
the system can retrieve the Leave Policy chunk rather than the entire handbook.
Why Is Chunking Needed in RAG?
Suppose you have a 200-page PDF.
If you create only one embedding for the entire PDF, that single vector must somehow represent:
- Introduction
- Pricing
- Refund policy
- Product features
- Technical documentation
- Legal terms
- Contact details
This creates a weak representation.
A user searching for:
What is the refund policy?
may not retrieve the correct information accurately because the whole document contains too many unrelated topics.
Chunking solves this problem.
Instead of:
200-page PDF
↓
1 Embedding
you create:
200-page PDF
↓
Many Smaller Chunks
↓
Embedding for Each Chunk
Now retrieval becomes much more precise.
Basic Chunking Workflow
A typical RAG indexing pipeline looks like:
Document
↓
Extract Text
↓
Clean Text
↓
Split into Chunks
↓
Create Embeddings
↓
Store Chunks + Vectors
When the user asks something:
Question
↓
Create Query Embedding
↓
Search Stored Chunk Vectors
↓
Retrieve Relevant Chunks
↓
Send Chunks to LLM
↓
Generate Answer
Chunking happens before the embeddings are created.
Simple Chunking Example
Suppose a document says:
Python is a programming language used for web development,
automation, data analysis, machine learning, and AI.
Flutter is a framework used to build cross-platform mobile,
web, and desktop applications.
RAG combines information retrieval with large language models.
You could split it into:
Chunk 1:
Python is a programming language used for web development,
automation, data analysis, machine learning, and AI.
Chunk 2:
Flutter is a framework used to build cross-platform mobile,
web, and desktop applications.
Chunk 3:
RAG combines information retrieval with large language models.
Now a query:
Which technology is used for AI development?
can retrieve Chunk 1.
A query:
What framework can build cross-platform applications?
can retrieve Chunk 2.
What Is Chunk Size?
Chunk size determines how much text is placed inside each chunk.
For example:
Chunk size = 100 words
means approximately every 100 words become a chunk.
Or you may use:
Chunk size = 500 tokens
Token-based chunking is common in LLM applications because language models operate on tokens.
Small Chunks vs Large Chunks
Chunk size directly affects retrieval quality.
Small Chunks
Example:
50 words per chunk
Advantages:
- More precise retrieval
- Less unrelated text
- Smaller context sent to the LLM
Disadvantages:
- Important context can be lost
- Sentences or ideas may be separated
- More chunks must be stored
- More embeddings may be required
Large Chunks
Example:
1,000 words per chunk
Advantages:
- More context remains together
- Fewer embeddings are needed
- Useful for long explanations
Disadvantages:
- More irrelevant information may be retrieved
- Semantic meaning can become less focused
- More tokens are sent to the LLM
Why There Is No Perfect Chunk Size
There is no universal chunk size that works best for every application.
The ideal chunk size depends on:
- Type of document
- User questions
- Embedding model
- LLM context window
- Retrieval strategy
- Document structure
- Amount of information needed to answer
For example, technical documentation may benefit from smaller section-based chunks.
Legal contracts may require larger chunks because clauses often depend on nearby paragraphs.
What Is Chunk Overlap?
One major problem with simple chunking is that information may be split at chunk boundaries.
Imagine:
Chunk 1:
Customers may request a refund within
and:
Chunk 2:
14 days after receiving the product.
Neither chunk contains the complete idea.
To reduce this problem, RAG systems often use chunk overlap.
Chunk Overlap Example
Without overlap:
Chunk 1:
Words 1–100
Chunk 2:
Words 101–200
Chunk 3:
Words 201–300
With 20-word overlap:
Chunk 1:
Words 1–100
Chunk 2:
Words 81–180
Chunk 3:
Words 161–260
Some words appear in neighboring chunks.
This helps preserve context.
Why Chunk Overlap Is Useful
Overlap can help when:
- A sentence crosses chunk boundaries
- A paragraph continues into the next chunk
- Important context appears before the main answer
- Pronouns depend on previous sentences
For example:
The Premium plan includes priority support.
Customers also receive unlimited projects.
If these lines are split across chunks, overlap can help keep them together.
Too Much Overlap Is Also a Problem
More overlap is not always better.
Suppose:
Chunk size = 500 tokens
Overlap = 450 tokens
Most of the content will be duplicated.
This can cause:
- More embeddings
- Higher storage usage
- Duplicate retrieval results
- Increased indexing cost
- Repeated information in the LLM context
Overlap should preserve useful continuity without excessive duplication.
Simple Python Chunking Function
A beginner-friendly word-based chunking function:
def chunk_text(text, chunk_size=100):
words = text.split()
chunks = []
for i in range(0, len(words), chunk_size):
chunk = " ".join(
words[i:i + chunk_size]
)
chunks.append(chunk)
return chunks
Example:
text = """
Python is a popular programming language...
"""
chunks = chunk_text(
text,
chunk_size=100
)
print(chunks)
Python Chunking with Overlap
A better version:
def chunk_text(
text,
chunk_size=100,
overlap=20
):
words = text.split()
chunks = []
start = 0
while start < len(words):
end = start + chunk_size
chunk = " ".join(
words[start:end]
)
chunks.append(chunk)
start += chunk_size - overlap
return chunks
This creates overlapping chunks.
Word-Based Chunking
Word-based chunking splits documents by word count.
Example:
Every 200 words
Advantages:
- Easy to implement
- Predictable
- Good for beginner projects
Disadvantages:
- Can split paragraphs unnaturally
- Does not understand document structure
- Token count may differ from word count
Character-Based Chunking
Character-based chunking splits by number of characters.
Example:
1,000 characters per chunk
This is also easy to implement.
However, characters do not correspond directly to semantic units or LLM tokens.
It is mainly useful for simple systems.
Token-Based Chunking
Token-based chunking splits according to tokens.
For example:
Chunk size = 500 tokens
Overlap = 50 tokens
This is useful because:
- Embedding models have token limits
- LLMs use tokens
- Cost is usually related to tokens
- Context windows are measured in tokens
Token-based chunking is generally more predictable for AI systems than word counts.
Sentence-Based Chunking
Sentence-based chunking preserves sentence boundaries.
Instead of cutting after exactly 100 words, you keep complete sentences.
For example:
Sentence 1
Sentence 2
Sentence 3
Sentence 4
could become:
Chunk 1:
Sentence 1 + Sentence 2
Chunk 2:
Sentence 3 + Sentence 4
This produces more natural chunks.
Paragraph-Based Chunking
Paragraph-based chunking uses existing paragraph boundaries.
Example:
Paragraph 1 → Chunk 1
Paragraph 2 → Chunk 2
Paragraph 3 → Chunk 3
This works well when each paragraph contains one complete idea.
However, paragraphs can vary greatly in size.
Some may contain:
20 words
while others contain:
1,000 words
So paragraph chunking often needs size controls.
Section-Based Chunking
Structured documents often contain headings.
For example:
1. Introduction
2. Pricing
3. Refund Policy
4. Account Management
Instead of ignoring these headings, you can use them as chunk boundaries.
For example:
Chunk:
Heading: Refund Policy
Content:
Customers can request...
This is usually better than splitting randomly every few hundred words.
Semantic Chunking
Semantic chunking attempts to split text based on changes in meaning.
Suppose a document moves from:
Flutter Authentication
to:
Flutter Payments
A semantic chunking system can detect the topic change and create a new chunk.
Conceptually:
Related Sentences
↓
Same Chunk
Topic Changes
↓
New Chunk
This can create more meaningful units.
Fixed-Size vs Semantic Chunking
Fixed-size chunking:
Every 500 tokens
is simple and predictable.
Semantic chunking:
Split when topic changes
can preserve meaning more naturally.
However, semantic chunking is more complex and may require additional processing.
Recursive Chunking
Recursive chunking tries larger natural separators first.
For example:
Document
↓
Split by Sections
↓
If section is too large:
Split by Paragraphs
↓
If paragraph is too large:
Split by Sentences
↓
If still too large:
Split by Tokens
This approach tries to preserve structure as much as possible.
It is commonly useful for RAG applications.
Document-Aware Chunking
Different documents have different structures.
For example, Markdown contains:
# Heading
## Subheading
Paragraph
HTML contains:
<h1>
<p>
<ul>
PDF documents may contain:
Page
Heading
Paragraph
Table
Document-aware chunking uses these structural signals.
This is often better than blindly splitting text into equal lengths.
Chunking PDFs
PDF chunking can be more challenging because extracted text may contain:
- Broken lines
- Headers
- Footers
- Tables
- Page numbers
- Multiple columns
A basic PDF pipeline:
PDF
↓
Extract Page Text
↓
Clean
↓
Chunk
↓
Store Page Metadata
A chunk can include:
{
"page": 12,
"text": "Customers may request refunds within 14 days..."
}
Storing page information makes citations possible later.
Chunking a PDF by Page
The simplest strategy is:
Page 1 → Chunk
Page 2 → Chunk
Page 3 → Chunk
This is easy but not always ideal.
A page may:
- Contain multiple unrelated topics
- Continue a section from the previous page
- Have too little content
Therefore, page boundaries are not always good semantic boundaries.
Better PDF Chunking
A better strategy might be:
PDF
↓
Extract Text + Page Metadata
↓
Detect Headings/Paragraphs
↓
Create Reasonable Chunks
↓
Preserve Page Numbers
This gives better retrieval while still allowing source references.
Chunking Tables
Tables require special care.
Imagine:
Plan | Price | Users
Basic | ₹499 | 5
Pro | ₹999 | 20
If you split this randomly, you may separate a row from its headers.
A better chunk preserves:
Plan: Basic
Price: ₹499
Users: 5
or preserves the complete small table.
Table-aware chunking can significantly improve retrieval accuracy.
Chunking Code
Code should also preserve logical structure.
Instead of:
Every 500 characters
you may split by:
- Function
- Class
- Method
- Module
For example:
def login():
...
should ideally remain together in one chunk.
This helps semantic code search and coding assistants.
Chunking Conversations
Chat histories can be chunked by:
- Message groups
- Topic changes
- Time windows
- Conversation turns
For example:
User Question
Assistant Answer
User Follow-Up
may be kept together because the messages depend on each other.
What Makes a Good Chunk?
A good chunk should be:
Self-Contained
The chunk should make sense without requiring too much missing context.
Bad:
This also applies for 14 days.
Good:
The product return policy allows returns within 14 days of delivery.
Focused
A chunk should ideally focus on one topic.
Bad:
Refund Policy
+
Company History
+
Pricing
+
Technical Setup
Good:
Refund Policy
Large Enough to Preserve Context
A chunk should include enough surrounding information to answer questions accurately.
Small Enough to Retrieve Precisely
The chunk should not contain so much unrelated content that its semantic meaning becomes diluted.
Include Useful Metadata
Store information such as:
Document Name
Page
Heading
Section
Category
URL
User ID
This improves filtering and citations.
Chunk Size Example
Suppose a user asks:
How long is the return period?
Your document contains:
Returns
Customers may return unused products within
14 days of receiving their order.
Products must remain in their original packaging.
A good chunk might be:
Returns
Customers may return unused products within
14 days of receiving their order.
Products must remain in their original packaging.
This preserves enough information to answer accurately.
Example of a Chunk That Is Too Small
within 14 days
This has very little context.
Is it:
- Return period?
- Payment period?
- Cancellation period?
The embedding may not represent the intended meaning clearly.
Example of a Chunk That Is Too Large
Imagine the same chunk also contains:
Company history
Shipping policy
Privacy policy
Terms of service
Refund policy
Payment rules
The relevant sentence is hidden among many unrelated concepts.
Retrieval can become less precise.
How Chunking Affects Embeddings
Each chunk becomes an embedding.
Chunk
↓
Embedding Model
↓
Vector
The vector represents the content of that chunk.
If a chunk contains one focused topic, the embedding can usually represent that topic more clearly.
If it contains many unrelated topics, the vector becomes a mixed representation.
This is why chunking and embeddings are closely connected.
How Chunking Affects Semantic Search
Suppose the query is:
How do I reset my password?
A good chunk:
Password Reset
Click "Forgot Password" on the login page.
A reset link will be sent to your email.
will likely have high semantic similarity.
A huge chunk containing:
Login
Payments
Notifications
Privacy
Account Settings
Password Reset
may be less precisely matched.
How Chunking Affects RAG Answers
RAG quality depends heavily on retrieval quality.
The flow is:
Chunking
↓
Embeddings
↓
Retrieval
↓
Context
↓
LLM Answer
If chunking is poor:
Poor Chunks
↓
Wrong Retrieval
↓
Wrong Context
↓
Poor Answer
A powerful LLM cannot reliably answer from information that was never retrieved.
Retrieval Failure Caused by Chunking
Suppose the correct answer is split across two chunks.
Chunk 1:
The customer can cancel the subscription
Chunk 2:
within 30 days without paying a fee.
The query:
Can I cancel without a fee?
may retrieve only one of them.
The answer becomes incomplete.
Overlap or structure-aware chunking can help avoid this.
Parent-Child Chunking
An advanced strategy is parent-child chunking.
You create:
Small Child Chunks
for precise retrieval.
But when one is matched, you return a larger:
Parent Section
to the LLM.
For example:
Parent:
Entire Refund Policy
Children:
Refund Time Limit
Refund Conditions
Non-Refundable Items
Refund Processing
Search uses the smaller child chunks.
The LLM receives the larger parent section.
This can combine precise search with better context.
Small-to-Big Retrieval
A similar strategy is:
Search Small Chunk
↓
Find Match
↓
Expand Context Around Match
↓
Send Larger Context to LLM
This is useful when a small match alone does not contain enough information.
Metadata-Aware Chunking
Each chunk can contain metadata:
{
"document": "employee_handbook.pdf",
"section": "Leave Policy",
"page": 18,
"year": 2026,
"text": "Employees receive..."
}
Then retrieval can use both:
Vector Similarity
+
Metadata Filters
For example:
Search:
remote work policy
Filter:
document_year = 2026
Chunk Titles Can Improve Retrieval
Suppose the raw content is:
Employees may request up to three days...
Without context, this could mean anything.
Add the heading:
Sick Leave Policy
Employees may request up to three days...
Now the embedding has much more context.
Including section headings in chunks can improve retrieval.
Should Headings Be Repeated in Every Chunk?
If one section is split into multiple chunks, repeating the heading can be useful.
Example:
Chunk 1:
Refund Policy
...
Chunk 2:
Refund Policy
...
Chunk 3:
Refund Policy
...
This helps each chunk remain understandable independently.
Chunking and Top-K Retrieval
Suppose you retrieve:
Top 5 chunks
If chunks are too small, Top 5 may not provide enough context.
If chunks are too large, Top 5 may provide far too much text.
Chunk size and top_k should therefore be tuned together.
Example
Small chunks:
200 tokens
Top K = 8
Large chunks:
800 tokens
Top K = 3
Either may work depending on the application.
There is no universal setting.
Chunking and LLM Context Windows
Retrieved chunks eventually go into the LLM context.
Suppose:
5 chunks
×
1,000 tokens each
=
5,000 tokens
If you retrieve:
20 chunks
you could send:
20,000 tokens
even before including:
- User question
- Instructions
- Conversation history
Chunking therefore affects token usage and cost.
Chunking and Cost
More chunks usually mean:
- More embeddings
- More vector records
- More storage
- More indexing operations
Larger chunks may mean:
- Fewer embeddings
- More context per retrieval
- Higher generation token usage
There is a trade-off.
The goal is not simply to minimize chunk count.
The goal is to maximize useful retrieval quality at a reasonable cost.
Chunking and Re-Ranking
A more advanced retrieval system might do:
Question
↓
Vector Search
↓
Top 20 Chunks
↓
Re-Ranker
↓
Best 5 Chunks
↓
LLM
Good chunking still matters because re-ranking cannot repair badly split information perfectly.
Chunking and Hybrid Search
Hybrid search combines:
Vector Search
+
Keyword Search
Chunking affects both.
For example, an exact error code:
ERR_AUTH_403
should remain in the same chunk as the explanation.
If chunking separates the code from its meaning, search quality suffers.
How to Choose a Chunk Size
Start by considering the type of question users will ask.
If users ask for specific facts:
What is the cancellation period?
smaller focused chunks may work well.
If users ask broad questions:
Explain the entire refund process.
larger chunks may provide better context.
Test with real questions instead of choosing a size only from online recommendations.
Practical Starting Point
For many text-heavy beginner RAG projects, you might begin with moderate chunks and a small overlap.
For example:
Chunk Size:
A few hundred tokens
Overlap:
Roughly 10% to 20%
Then evaluate real search results and adjust.
These are starting ideas, not universal rules.
How to Evaluate Chunking
Create questions whose answers you already know.
For example:
Question:
How many annual leave days are provided?
Correct Section:
Leave Policy
Run retrieval and check:
Did the correct chunk appear?
Was it in the top results?
Did the chunk contain the complete answer?
Was unnecessary information included?
Repeat this across many realistic questions.
Retrieval Evaluation Example
Create:
Question 1
Expected Chunk
Question 2
Expected Chunk
Question 3
Expected Chunk
Then measure:
Top-1 Retrieval Accuracy
Top-3 Retrieval Accuracy
Top-5 Retrieval Accuracy
This gives you actual evidence about whether your chunking strategy works.
Common Chunking Mistakes
Using One Chunk for the Entire Document
This usually reduces retrieval precision for long documents.
Making Chunks Extremely Small
Tiny chunks often lose important context.
Making Chunks Extremely Large
Large chunks may mix unrelated topics.
Ignoring Document Structure
Headings and paragraphs are valuable signals.
No Overlap When Context Crosses Boundaries
Important ideas can be split.
Excessive Overlap
This creates unnecessary duplication.
Removing Important Metadata
Source information helps filtering and citations.
Splitting Tables Randomly
This can destroy relationships between values.
Splitting Code in the Middle of Functions
This reduces code-search usefulness.
Fixed-Size Chunking vs Recursive Chunking
Fixed-size:
Every 500 tokens
Advantages:
- Simple
- Fast
- Predictable
Recursive:
Section
↓
Paragraph
↓
Sentence
↓
Token
Advantages:
- Preserves natural structure
- Often produces more readable chunks
For structured content, recursive approaches are often worth considering.
Chunking vs Embedding
These concepts are different.
Chunking:
Splits Text
Embedding:
Converts Chunk into Vector
The workflow is:
Document
↓
Chunking
↓
Chunks
↓
Embeddings
↓
Vectors
Chunking vs Semantic Search
Chunking prepares the searchable units.
Semantic search finds the chunks relevant to the user’s query.
Document
↓
Chunking
↓
Embeddings
↓
Semantic Search
Chunking vs Vector Database
Chunking decides what text units are stored.
The vector database stores and searches their embeddings.
Chunk
↓
Embedding
↓
Vector Database
Chunking vs RAG
Chunking is one stage in a larger RAG system.
Document
↓
Chunking
↓
Embedding
↓
Vector Storage
↓
Retrieval
↓
LLM
↓
Answer
Poor chunking can negatively affect the entire pipeline.
Example Production Chunk Record
A production chunk might look like:
{
"chunk_id": "policy_18_02",
"document_id": "employee_handbook",
"title": "Annual Leave Policy",
"page": 18,
"text": "Employees receive 24 paid annual leave days...",
"embedding": [0.12, -0.38, 0.71]
}
This gives the system:
- Searchable meaning
- Original text
- Source document
- Page number
- Section title
Chunking for Multiple Documents
If you have:
PDF 1
PDF 2
PDF 3
PDF 4
you can chunk all of them and store metadata.
Example:
{
"file": "refund_policy.pdf",
"page": 4,
"section": "Refund Period",
"text": "Customers may request..."
}
When retrieved, your chatbot can say:
Source:
refund_policy.pdf
Page 4
Chunking for Frequently Updated Data
Suppose only one section changes.
Instead of reprocessing the entire knowledge base, you can update only affected chunks.
For example:
Product Pricing Section
↓
Changed
↓
Re-Chunk Section
↓
Generate New Embeddings
↓
Replace Old Chunks
This makes maintenance easier.
Chunk Deduplication
Sometimes documents contain repeated text such as:
Company Confidential
on every page.
If every chunk includes this repeatedly, retrieval may become noisy.
You can remove unnecessary:
- Headers
- Footers
- Navigation
- Repeated disclaimers
- Duplicate text
before chunking.
Preserve Important Context
Do not remove information just because it looks repetitive if it is necessary for meaning.
For example, repeating:
Refund Policy
as a section heading may actually improve retrieval.
Cleaning should remove noise, not useful context.
Chunking Best Practices
A good RAG chunking strategy generally follows these principles:
- Preserve meaningful sections whenever possible.
- Keep chunks focused on a small number of related ideas.
- Use enough overlap to preserve context across boundaries.
- Avoid excessive duplication.
- Store source metadata with every chunk.
- Keep titles and headings when they help explain the content.
- Treat tables, code, and structured data carefully.
- Test using real user questions.
- Tune chunk size together with retrieval settings.
- Evaluate retrieval before blaming the LLM.
Frequently Asked Questions
What is chunking in RAG?
Chunking is the process of splitting large documents into smaller sections before creating embeddings and storing them for retrieval.
Why is chunking important?
It improves retrieval precision by allowing a RAG system to search smaller, more focused pieces of information.
What is chunk size?
Chunk size determines how much text is included in each chunk, usually measured in words, characters, or tokens.
What is chunk overlap?
Chunk overlap means repeating some text between neighboring chunks to preserve context across chunk boundaries.
What is the best chunk size for RAG?
There is no universal best size. It depends on the document, queries, embedding model, retrieval strategy, and desired context.
Is smaller chunking always better?
No. Very small chunks can lose important context.
Are larger chunks better?
Not always. Very large chunks can contain too much unrelated information and reduce retrieval precision.
Should chunks overlap?
Overlap is often useful, especially for continuous prose, but too much overlap causes duplication.
Can I chunk PDFs by page?
Yes, but page boundaries are not always semantic boundaries. Structure-aware chunking is often better.
What is semantic chunking?
Semantic chunking splits documents based on meaning or topic changes instead of using only fixed sizes.
What is recursive chunking?
Recursive chunking tries to split by natural boundaries such as sections, paragraphs, and sentences before falling back to smaller units.
Does chunking affect RAG accuracy?
Yes. Chunking strongly affects embedding quality, retrieval results, context quality, and ultimately the final LLM answer.
Final Thoughts
Chunking may look like a simple preprocessing step, but it is one of the most important parts of a RAG system.
The overall flow is:
Document
↓
Chunking
↓
Chunks
↓
Embeddings
↓
Vector Search
↓
Relevant Chunks
↓
LLM
↓
Answer
The objective is not simply to make chunks small.
The objective is to create chunks that are:
Focused
Self-Contained
Meaningful
Searchable
Context-Rich
A good chunk should contain enough information to preserve meaning while remaining small enough for precise retrieval.
For simple projects, you can start with:
Fixed-size chunks
+
Small overlap
As the application becomes more advanced, you can explore:
Sentence-based chunking
Section-based chunking
Recursive chunking
Semantic chunking
Parent-child retrieval
Metadata-aware chunking
Table-aware chunking
Code-aware chunking
When RAG results are poor, do not immediately assume the LLM is the problem.
Very often, the real issue is earlier in the pipeline:
Bad Chunking
↓
Bad Retrieval
↓
Bad Context
↓
Bad Answer
Understanding chunking properly is therefore essential for building reliable RAG applications, PDF chatbots, AI knowledge assistants, semantic search systems, and document-based AI applications.
