Filtering out content that degrades a RAG application's quality

Before chunking and embedding documents for a RAG application, you must remove content that adds noise without adding meaning, such as boilerplate, navigation text, and formatting artifacts, because this content pollutes embeddings and retrieval results. Databricks recommends cleaning and filtering source documents as an explicit step in the data preparation pipeline, typically implemented with PySpark UDFs or standard Python parsing/regex logic within a Delta Live Tables or notebook workflow.

1 · Learn the must-know

• 1Extraneous content such as headers, footers, page numbers, HTML/XML tags, disclaimers, boilerplate legal text, and repeated navigation menus should be stripped because it dilutes the semantic signal in each chunk and can dominate embedding similarity.
• 2Filtering is typically done with custom Python/PySpark functions (e.g., regex-based cleaning UDFs) applied during ingestion, before text is split into chunks and passed to an embedding model.
• 3Low-quality or irrelevant documents (duplicates, empty files, non-target languages, corrupted parses) should be filtered out at the document level, not just cleaned at the text level, to avoid wasting index space and degrading retrieval precision.
• 4Poorly parsed content from source formats like PDFs (e.g., broken tables, garbled OCR text, repeated whitespace) is a common quality issue and should be validated/cleaned as part of the parsing step, not left for the LLM to interpret.
• 5Overly generic or short chunks left after filtering (e.g., a chunk that is just a page number or a table of contents entry) provide little retrieval value and can be filtered out based on minimum length or content heuristics.
• 6Cleaning and filtering logic should be applied consistently and stored as reusable pipeline steps (e.g., in Delta tables) so the same transformations can be reapplied when source documents are updated, keeping the vector index and source data in sync.

2 · Check your understanding