Identifying where data comes from and where it needs to land
Defining data sources and sinks means identifying where data originates (e.g., Pub/Sub, Cloud Storage, on-prem databases, streaming apps) and where it lands after processing (e.g., BigQuery, Bigtable, Cloud Storage, Cloud SQL) so pipelines can be designed with correct connectors, schemas, and throughput expectations. Choosing the right source/sink pair drives decisions about pipeline tooling (Dataflow, Dataproc, Data Fusion) and affects latency, cost, and scalability. This is foundational to designing Apache Beam/Dataflow pipelines and other ETL/ELT workflows on Google Cloud.
1 · Learn the must-know
- Pub/Sub is the standard source/sink for streaming ingestion and supports at-least-once delivery with ordering keys for sequencing when needed.
- Cloud Storage is commonly used as a staging sink for batch data and as a durable landing zone before loading into BigQuery or triggering downstream processing via Eventarc/Cloud Functions.
- BigQuery is a frequent sink for analytics workloads and supports both batch loads and streaming inserts, but streaming inserts have different cost and quota behavior than load jobs.
- Dataflow (Apache Beam) natively supports many source/sink connectors (Pub/Sub, BigQuery, Cloud Storage, JDBC, Bigtable) via I/O transforms, and custom sources/sinks can be built using the Beam SDK if a connector doesn't exist.
- When defining sinks, always consider schema compatibility, write disposition (append/overwrite/truncate), and idempotency to avoid duplicate or lost data during retries.
- Datastream and Database Migration Service are used for CDC-based ingestion from operational databases (MySQL, PostgreSQL, Oracle) into sinks like BigQuery or Cloud Storage without invoking custom pipeline code.
2 · Check your understanding
A team builds a streaming Dataflow pipeline that reads order events from a Pub/Sub topic and writes aggregated totals to a BigQuery table using streaming inserts. During testing, a worker crash causes the pipeline to retry a bundle of records, and the team notices duplicate rows appearing in the destination table. Which change to the sink configuration will prevent this duplication on retries?
What you have tried across GCP PDE's objectives, not a readiness score.
Designing data processing systems~22% of the exam0 of 17 tried
Ingesting and processing the data~25% of the exam0 of 11 tried
Storing the data~20% of the exam0 of 13 tried
Preparing and using data for analysis~15% of the exam0 of 11 tried
Maintaining and automating data workloads~18% of the exam0 of 15 tried
3 · Keep going
Ready for more? Take a weighted mock or try free practice questions.