Picking a chunking strategy for a document structure and model's constraints

Chunking splits documents into passages sized to fit within an embedding model's and LLM's context window while preserving enough semantic coherence for accurate retrieval. The right chunk size and overlap depend on document structure (e.g., code, markdown, PDFs, transcripts) and downstream model token limits, since chunks that are too large lose retrieval precision and chunks that are too small lose context.

1 · Learn the must-know

• 1Chunk size must respect the embedding model's max input token limit, not just the LLM's context window, since embeddings are computed per chunk.
• 2Structure-aware splitting (by headers, paragraphs, sentences, or code functions) generally outperforms fixed-length character or token splitting because it preserves semantic units.
• 3Adding overlap between consecutive chunks helps prevent loss of context at chunk boundaries but increases storage and retrieval redundancy.
• 4Smaller chunks improve retrieval precision (more specific matches) but may lack surrounding context needed for the LLM to generate a complete answer.
• 5Larger chunks provide more context per retrieval but risk exceeding token limits when multiple chunks are concatenated into the prompt, and can dilute embedding relevance.
• 6Different document types (structured tables, code, long-form prose, transcripts) require different chunking strategies rather than a one-size-fits-all approach.

2 · Check your understanding