Watching and coordinating a pipeline's moving parts

Data pipeline orchestration on Google Cloud is primarily handled by Cloud Composer (managed Apache Airflow), which schedules and coordinates dependencies across GCP services like Dataflow, BigQuery, and Dataproc. Monitoring relies on Cloud Monitoring and Cloud Logging for metrics, dashboards, alerting, and log-based troubleshooting across pipeline components.

Must-know

• Cloud Composer uses DAGs (Directed Acyclic Graphs) written in Python to define task dependencies and scheduling, running on a managed Google Kubernetes Engine cluster.
• Dataflow jobs automatically emit metrics to Cloud Monitoring (e.g., system lag, data freshness, throughput) without additional instrumentation, and these can be viewed in the Dataflow job monitoring UI or custom dashboards.
• Cloud Monitoring alerting policies can trigger notifications (via channels like email, Pub/Sub, or PagerDuty) when pipeline metrics breach thresholds, such as high system lag or failed job counts.
• Cloud Logging aggregates logs from Dataflow, Dataproc, BigQuery, and Composer, and supports log-based metrics and sinks for exporting logs to BigQuery or Cloud Storage for further analysis.
• A common gotcha is that Composer environments have their own Airflow web UI and scheduler health metrics separate from the DAG task-level logs, so both environment health and task execution must be monitored independently.
• For event-driven or serverless orchestration without needing full Airflow capabilities, Cloud Functions or Workflows can trigger and chain pipeline steps based on events like new files in Cloud Storage or Pub/Sub messages.

Coverage checked against the published exam guide on Aug 11, 2026.

These are independent practice questions, written against this certification's published exam guide. They are not the certification vendor's own questions, and not the real exam.