Picking the right ingestion or processing service for a given source

The GCP data engineer exam expects you to map each ingestion/processing tool to its ideal use case based on data volume, latency, transformation complexity, and existing skillsets. Choosing between managed serverless services (Dataflow, BigQuery, Pub/Sub) versus cluster-based services (Dataproc) is a recurring exam theme, especially around migration and cost scenarios. Recognize open-source lineage (Apache Beam, Spark, Hadoop, Kafka) since Google's managed services are often built on or compatible with these frameworks.

Must-know

  • Dataflow is Google's serverless, autoscaling runner for Apache Beam pipelines, supporting unified batch and streaming with no cluster management, ideal for new pipeline development.
  • Dataproc runs managed Hadoop/Spark clusters, best when migrating existing Spark/Hadoop/Hive/Pig jobs with minimal code changes or when you need fine-grained cluster/version control; supports ephemeral clusters for cost savings.
  • Pub/Sub is the entry point for streaming ingestion, providing globally durable, at-least-once delivery messaging that decouples producers from consumers before processing (often feeding Dataflow).
  • Cloud Data Fusion is a fully managed, visual/GUI-based ETL tool (built on CDAP) targeted at users who want code-free pipeline building and reuse of pre-built connectors/transformations rather than writing Beam/Spark code.
  • BigQuery supports direct data ingestion (batch loads, streaming inserts, or federated/external queries) and can itself perform transformation via SQL (ELT pattern), sometimes eliminating the need for a separate processing engine.
  • Kafka is not a native GCP managed service on the exam by default, Google's equivalent/managed alternative is Pub/Sub, though self-managed Kafka on Compute Engine/GKE or via partner solutions can still integrate with Dataflow for ingestion.
Drill this objectiveTake a weighted mockTry free practice questions

Coverage checked against the published exam guide on Aug 11, 2026.

These are independent practice questions, written against this certification's published exam guide. They are not the certification vendor's own questions, and not the real exam.