Building a CI/CD path for pipeline code changes
CI/CD for data pipelines on GCP means automating the build, test, and deployment of ingestion and processing artifacts (Dataflow templates, Dataproc jobs, dbt/SQL, Cloud Composer DAGs, Cloud Functions) using Cloud Build, Artifact Registry, and Cloud Source Repositories or GitHub/GitLab integrations. The goal is repeatable, version-controlled deployment across dev/test/prod environments with automated validation before promotion.
Must-know
- Cloud Build is the core CI/CD orchestrator on GCP, triggered by commits to Cloud Source Repositories, GitHub, or Bitbucket via triggers defined in a cloudbuild.yaml.
- Dataflow supports Flex Templates, which package pipeline code and dependencies into a container image stored in Artifact Registry, enabling versioned, parameterized deployment via CI/CD without re-uploading source.
- Cloud Composer DAGs are deployed via CI/CD by syncing Python files to the environment's GCS bucket (dags/ folder); a common pattern tests DAGs (e.g., with pytest and DAG integrity checks) before syncing to production.
- Artifact Registry stores container images and language packages (Python, Maven) used by Dataflow templates, Dataproc custom images, and Cloud Functions, and integrates natively with Cloud Build triggers.
- Terraform (via Cloud Build or Cloud Deploy) is the standard IaC approach for provisioning data infra (BigQuery datasets, Pub/Sub topics, Dataproc clusters), and should be part of the same CI/CD pipeline as code deployment for consistency.
- A common exam gotcha: separate environments (dev/staging/prod) typically use separate GCP projects, with CI/CD promoting artifacts across projects rather than deploying directly to prod, requiring appropriate service account IAM permissions per stage.
Coverage checked against the published exam guide on Aug 11, 2026.
These are independent practice questions, written against this certification's published exam guide. They are not the certification vendor's own questions, and not the real exam.