Fixing messy records before they reach a report

Data cleaning on Google Cloud can be performed with code-free tools like Cloud Data Fusion (via Wrangler and prebuilt transformation directives) or with code-based approaches using BigQuery SQL for in-warehouse cleaning and Dataflow for large-scale, pipelined batch/streaming transformations. The Associate Data Practitioner exam expects you to know which tool fits a given cleaning scenario based on skill level, data volume, and whether processing is batch or streaming.

1 · Learn the must-know

• 1Cloud Data Fusion's Wrangler provides a visual, spreadsheet-like interface with prebuilt 'directives' (e.g., parse, deduplicate, mask, fill null) for non-engineers to clean data without writing code.
• 2BigQuery supports data cleaning directly in SQL using functions like TRIM, CAST, IFNULL/COALESCE, REGEXP_REPLACE, and DISTINCT/QUALIFY for deduplication, ideal when data is already loaded into BigQuery.
• 3Dataflow (based on Apache Beam) is the preferred choice for large-scale, complex, or streaming data cleaning/transformation pipelines that need custom logic beyond SQL, and it scales automatically.
• 4Cloud Data Fusion pipelines can include a Wrangler transform stage that generates a recipe of directives, which can then be reused and scheduled as part of a full ETL/ELT pipeline.
• 5BigQuery's built-in data quality/scheduled queries or MERGE statements are commonly used to deduplicate, standardize, or update data incrementally without needing a separate ETL tool.
• 6For streaming or near-real-time cleaning (e.g., filtering malformed records before landing in BigQuery), Dataflow is generally favored over Cloud Data Fusion or plain SQL, which are better suited to batch/ELT-style cleaning.

2 · Check your understanding