Designing a system that tolerates failure and restarts cleanly

Fault-tolerant data workloads on Google Cloud rely on idempotent processing, checkpointing, and automatic retry/restart mechanisms built into services like Dataflow, Dataproc, and Cloud Composer. Designing for fault tolerance means anticipating worker failures, transient errors, and job restarts without data loss or duplication. Understanding each service's native restart and recovery semantics is critical for the exam.

1 · Learn the must-know

• 1Dataflow automatically retries failed work items and uses windowing with checkpointing to recover state after worker failures, but pipelines must be designed with idempotent side effects since retries can reprocess data.
• 2Dataproc supports high-availability mode with multiple master nodes and can use preemptible/spot secondary workers, but jobs on preemptible VMs must handle sudden termination and checkpoint intermediate state to Cloud Storage or HDFS.
• 3Cloud Composer (Apache Airflow) allows configuring retries, retry_delay, and on_failure_callback at the task level, and failed DAG runs can be cleared or backfilled to resume from the point of failure rather than restarting entirely.
• 4For streaming pipelines, using exactly-once processing semantics in Dataflow combined with deduplication logic (e.g., unique record IDs) protects against duplicate processing after restarts.
• 5Dead-letter queues (via Pub/Sub subscriptions or custom Dataflow sinks) should be used to isolate poison messages so a single bad record doesn't crash or stall an entire pipeline.
• 6Designing for graceful degradation, such as using Cloud Monitoring alerts and Cloud Functions/Cloud Composer sensors to detect stuck jobs, enables automated restart workflows rather than relying solely on manual intervention.

2 · Check your understanding