Processing data once it is sitting in storage
Processing data for storage on Google Cloud means choosing the right compute service to transform, clean, and load data into your target store, Cloud Storage, BigQuery, Bigtable, Spanner, or Firestore, using either batch or streaming pipelines. The exam tests your ability to match a workload's latency, scale, and schema requirements to Dataflow, Dataproc, BigQuery, or Data Fusion, and to understand how processing choices affect downstream storage design (partitioning, clustering, schema evolution).
1 · Learn the must-know
- Dataflow (Apache Beam) is the preferred serverless service for both batch and streaming pipelines and unifies the two under one programming model, making it the default choice when the exam mentions 'unified batch and streaming' or autoscaling ETL.
- Dataproc runs managed Hadoop/Spark clusters and is the right answer when a scenario requires migrating existing Spark/Hadoop jobs or needs fine-grained cluster control, not for greenfield serverless pipelines.
- BigQuery can process data in place via SQL (ELT) using scheduled queries, materialized views, or BigQuery ML, so heavy transformation logic doesn't always need an external pipeline, prefer ELT when data is already in BigQuery and transformations are SQL-expressible.
- Cloud Data Fusion is a GUI-based, no-code/low-code ETL tool built on CDAP, useful when the exam scenario emphasizes visual pipeline design or data governance/lineage for less technical teams.
- Streaming inserts into BigQuery (via the Storage Write API or legacy streaming API) incur different cost/latency tradeoffs than batch loads, and small/frequent writes can hit quota limits if not batched appropriately.
- Pub/Sub is the standard ingestion buffer feeding Dataflow streaming pipelines before data lands in Bigtable, BigQuery, or Cloud Storage, decoupling producers from processing to handle bursty or out-of-order data with watermarks and windowing.
2 · Check your understanding
A retail company streams point-of-sale transactions through a Pub/Sub topic into a Dataflow streaming pipeline that computes five-minute revenue totals per store. Because of intermittent store network outages, some transaction events arrive several minutes after their event timestamp. The team wants late transactions to still be reflected in the correct five-minute total instead of being silently discarded. What should they configure in the pipeline?
What you have tried across GCP PDE's objectives, not a readiness score.
Designing data processing systems~22% of the exam0 of 17 tried
Ingesting and processing the data~25% of the exam0 of 11 tried
Storing the data~20% of the exam0 of 13 tried
Preparing and using data for analysis~15% of the exam0 of 11 tried
Maintaining and automating data workloads~18% of the exam0 of 15 tried
3 · Keep going
Ready for more? Take a weighted mock or try free practice questions.