Separating development and production environments cleanly

Google Cloud data processing designs must support distinct development and production environments so teams can iterate quickly without risking production data quality, cost, or availability. This typically involves separate GCP projects, environment-specific configurations, and controlled promotion paths from dev to test to production.

1 · Learn the must-know

• 1Use separate GCP projects (not just datasets or folders) for dev, test, and prod to get clean IAM boundaries, billing isolation, and quota separation.
• 2Dataflow, Dataproc, and Composer support environment-specific parameters (e.g., pipeline options, Airflow variables, Terraform workspaces) so the same code/DAG can run against different service accounts, datasets, and cluster sizes per environment.
• 3Production environments should favor autoscaling and managed services (e.g., Dataproc autoscaling policies, Dataflow autoscaling) while dev/test can use smaller fixed-size or ephemeral clusters to control cost.
• 4Use BigQuery datasets or projects labeled by environment plus row-level/column-level security and separate service accounts so dev engineers cannot accidentally read or write production data.
• 5CI/CD pipelines (Cloud Build, Cloud Deploy) should promote artifacts (container images, JAR files, DAGs) through dev, staging, and prod using the same build artifact to avoid environment drift.
• 6Cost controls differ by environment: dev/test often use preemptible/Spot VMs and smaller quotas, while production uses committed use discounts and SLA-backed managed services for reliability.

2 · Check your understanding