Defining what a pipeline transforms and in what order

Data transformation and orchestration on Google Cloud typically means designing pipelines that clean, join, and reshape data (via Dataflow, Dataproc, or Cloud Data Fusion) and then scheduling and sequencing those jobs with a workflow orchestrator like Cloud Composer. The exam tests whether you can pick the right transformation engine for batch vs. streaming needs and the right orchestration tool to manage dependencies, retries, and scheduling across GCP services.

1 · Learn the must-know

• 1Cloud Composer (managed Apache Airflow) is the primary orchestration service for defining DAGs that coordinate multi-step, cross-service pipelines (e.g., trigger Dataflow, then load BigQuery, then run a validation query) with dependency management, retries, and SLAs.
• 2Dataflow (Apache Beam) is the preferred choice for unified batch and streaming transformation logic, especially when you need windowing, watermarks, and autoscaling; Dataproc is preferred when you have existing Spark/Hadoop transformation code or need fine-grained cluster control.
• 3Cloud Data Fusion provides a visual, code-free interface for building ETL/ELT transformation pipelines and is suited for teams wanting reusable, GUI-based pipelines with built-in connectors rather than custom Beam/Spark code.
• 4Dataprep (Cloud Dataprep by Trifacta) is used for visual, exploratory data cleaning/wrangling before building a repeatable transformation pipeline, not for production orchestration.
• 5For simple, single-step scheduled transformations, Cloud Scheduler plus Cloud Functions/Cloud Run or BigQuery scheduled queries can be sufficient and cheaper than standing up Composer for lightweight orchestration needs.
• 6A common exam gotcha: orchestration tools (Composer) coordinate and schedule jobs across services, while transformation engines (Dataflow/Dataproc/Data Fusion) actually execute the data processing logic, confusing which layer a service belongs to is a frequent mistake.

2 · Check your understanding