Picking the right ingestion or processing service for a given source

The GCP data engineer exam expects you to map each ingestion/processing tool to its ideal use case based on data volume, latency, transformation complexity, and existing skillsets. Choosing between managed serverless services (Dataflow, BigQuery, Pub/Sub) versus cluster-based services (Dataproc) is a recurring exam theme, especially around migration and cost scenarios. Recognize open-source lineage (Apache Beam, Spark, Hadoop, Kafka) since Google's managed services are often built on or compatible with these frameworks.

1 · Learn the must-know

• 1Dataflow is Google's serverless, autoscaling runner for Apache Beam pipelines, supporting unified batch and streaming with no cluster management, ideal for new pipeline development.
• 2Dataproc runs managed Hadoop/Spark clusters, best when migrating existing Spark/Hadoop/Hive/Pig jobs with minimal code changes or when you need fine-grained cluster/version control; supports ephemeral clusters for cost savings.
• 3Pub/Sub is the entry point for streaming ingestion, providing globally durable, at-least-once delivery messaging that decouples producers from consumers before processing (often feeding Dataflow).
• 4Cloud Data Fusion is a fully managed, visual/GUI-based ETL tool (built on CDAP) targeted at users who want code-free pipeline building and reuse of pre-built connectors/transformations rather than writing Beam/Spark code.
• 5BigQuery supports direct data ingestion (batch loads, streaming inserts, or federated/external queries) and can itself perform transformation via SQL (ELT pattern), sometimes eliminating the need for a separate processing engine.
• 6Kafka is not a native GCP managed service on the exam by default, Google's equivalent/managed alternative is Pub/Sub, though self-managed Kafka on Compute Engine/GKE or via partner solutions can still integrate with Dataflow for ingestion.

2 · Check your understanding