Planning for recovery when a pipeline or region fails

Disaster recovery (DR) and fault tolerance for GCP data systems are governed by two key metrics: Recovery Time Objective (RTO) and Recovery Point Objective (RPO). Designing for these requires deliberate choices about data replication (multi-region vs. regional), backup strategies, and service redundancy across zones and regions. Google-managed services like BigQuery and Spanner offer built-in high availability, while services like Compute Engine or self-managed databases require explicit architectural decisions for resilience.

1 · Learn the must-know

• 1RTO measures how quickly a system must recover after a disruption, while RPO measures the maximum acceptable data loss measured in time (e.g., losing 5 minutes of data).
• 2Multi-region storage (e.g., GCS multi-region, BigQuery multi-region datasets) provides higher availability and durability but may increase cost and write latency compared to regional resources.
• 3Cloud SQL and Spanner support cross-region read replicas and automated backups; Cloud SQL failover requires configuring a standby instance, whereas Spanner's multi-region configurations provide automatic synchronous replication and failover.
• 4BigQuery automatically replicates data across zones within a region (and across regions for multi-region datasets) and manages failover transparently, but does not natively support customer-triggered geographic failover for regional datasets without manual dataset copy.
• 5For Dataflow and other compute-based pipelines, fault tolerance often means designing idempotent, checkpointed, and stateful processing so jobs can resume from failure without data loss or duplication.
• 6Pub/Sub provides at-least-once delivery with built-in message retention and redelivery, which supports fault-tolerant, decoupled pipeline design, but downstream consumers must handle potential duplicate messages.

2 · Check your understanding