Onboarding a data source the pipeline has not seen before
Integrating with new data sources on Google Cloud requires selecting the right ingestion pattern (batch, streaming, or change data capture) based on source type, volume, and latency needs, then choosing an appropriate connector or service to land data reliably. Google Cloud provides managed services like Datastream, Pub/Sub, Storage Transfer Service, BigQuery Data Transfer Service, and Dataflow templates to reduce custom integration code, while Cloud Data Fusion and Apiary/Apigee-style API integrations extend connectivity to SaaS and on-premises systems. Understanding source constraints (schema, network access, authentication) and target requirements (schema evolution, deduplication, exactly-once semantics) is key to designing a robust pipeline.
1 · Learn the must-know
- Datastream provides serverless change data capture (CDC) and replication from Oracle, MySQL, PostgreSQL, and AlloyDB into BigQuery, Cloud Storage, or Cloud SQL with minimal source impact.
- Pub/Sub is the standard entry point for streaming and event-driven integrations, decoupling producers and consumers and supporting push/pull delivery with at-least-once guarantees.
- Storage Transfer Service handles large-scale batch data movement from on-premises, other clouds (S3, Azure Blob), or HTTP/HTTPS sources into Cloud Storage, including scheduled and incremental transfers.
- BigQuery Data Transfer Service automates recurring loads from SaaS applications (Google Ads, YouTube, etc.) and other data warehouses (Teradata, Redshift, S3) directly into BigQuery.
- Cloud Data Fusion offers a visual, no/low-code interface with hundreds of pre-built connectors (via CDAP plugins) for integrating diverse enterprise and cloud data sources into pipelines.
- For custom or unsupported sources, Dataflow (with Apache Beam I/O connectors) or a Pub/Sub-based ingestion pattern combined with Cloud Functions/Cloud Run is the typical fallback for building bespoke connectors, and network connectivity (VPC peering, Private Service Connect, or Cloud VPN/Interconnect) must be considered for on-premises or hybrid sources.
2 · Check your understanding
A retailer runs a mission-critical PostgreSQL database on-premises and wants to integrate it as a new streaming data source for BigQuery, capturing inserts, updates, and deletes with minimal latency and without managing custom replication code. Which approach should the data engineer choose?
What you have tried across GCP PDE's objectives, not a readiness score.
Designing data processing systems~22% of the exam0 of 17 tried
Ingesting and processing the data~25% of the exam0 of 11 tried
Storing the data~20% of the exam0 of 13 tried
Preparing and using data for analysis~15% of the exam0 of 11 tried
Maintaining and automating data workloads~18% of the exam0 of 15 tried
3 · Keep going
Ready for more? Take a weighted mock or try free practice questions.