Shaping data during the ingestion pipeline itself
Data transformation on Google Cloud spans batch and streaming pipelines, with Dataflow (Apache Beam) as the unified serverless engine, alongside Dataproc for Spark/Hadoop-based ETL and Data Fusion for GUI-driven pipeline design. Choosing the right transformation tool and pattern depends on data volume, latency needs, existing skill sets, and whether transformations are ELT (in-warehouse, e.g., BigQuery SQL) or ETL (pre-load, e.g., Dataflow).
1 · Learn the must-know
- Apache Beam's unified model lets the same pipeline code run in both batch and streaming mode on Dataflow, simplifying reprocessing and code reuse.
- Dataflow autoscaling and dynamic work rebalancing handle uneven data distribution and late-arriving data using windowing, watermarks, and triggers for streaming transformations.
- BigQuery supports in-place ELT transformations via SQL, scheduled queries, and materialized views, avoiding the need to move data out for many transformation tasks.
- Dataproc is preferred when migrating existing Spark/Hadoop jobs with minimal code changes, but requires more cluster management than serverless options like Dataflow.
- Cloud Data Fusion provides a visual, plugin-based interface for building ETL/ELT pipelines without writing code, useful for teams without deep programming expertise.
- For side inputs, joins, or deduplication in streaming pipelines, careful windowing and state management in Beam/Dataflow are essential to avoid incorrect or incomplete results.
2 · Check your understanding
A retail company uses a Dataflow streaming pipeline to compute hourly total sales from a Pub/Sub topic. Due to mobile network delays, roughly 5% of events arrive up to 15 minutes after the watermark has passed their event time. The pipeline must correct the previously emitted hourly totals as this late data arrives, rather than dropping it or emitting duplicate totals. Which windowing configuration should the data engineer apply to the PCollection?
What you have tried across GCP PDE's objectives, not a readiness score.
Designing data processing systems~22% of the exam0 of 17 tried
Ingesting and processing the data~25% of the exam0 of 11 tried
Storing the data~20% of the exam0 of 13 tried
Preparing and using data for analysis~15% of the exam0 of 11 tried
Maintaining and automating data workloads~18% of the exam0 of 15 tried
3 · Keep going
Ready for more? Take a weighted mock or try free practice questions.