Choosing between a persistent cluster and a job-based one

For Dataproc, choose between persistent (long-running) clusters and ephemeral, job-scoped clusters based on workload patterns, cost sensitivity, and isolation needs. Ephemeral clusters are the Google-recommended default for most batch workloads because you pay only for the job's runtime and avoid idle capacity. Persistent clusters remain appropriate for interactive, low-latency, or continuously running workloads (e.g., Spark shells, notebooks, streaming) where cluster startup time is unacceptable.

1 · Learn the must-know

• 1Ephemeral clusters are created per job (often via Workflow Templates or an orchestrator like Cloud Composer) and deleted immediately after completion, minimizing cost since billing is per-second while nodes run.
• 2Persistent clusters incur continuous cost even when idle, so they must be actively monitored, autoscaled, or scheduled for start/stop to control spend.
• 3Storing data and metadata outside the cluster lifecycle (e.g., in Cloud Storage rather than HDFS, and using an external Hive metastore like Dataproc Metastore) is essential for ephemeral clusters so state isn't lost when the cluster is deleted.
• 4Job-based clusters provide strong workload isolation, each job gets a fresh environment, avoiding resource contention, dependency conflicts, and noisy-neighbor issues common on shared persistent clusters.
• 5Cluster startup time (typically 1-2+ minutes) is a real tradeoff for ephemeral clusters, making persistent clusters preferable for latency-sensitive interactive or streaming use cases.
• 6Dataproc Workflow Templates are the standard mechanism to define, parameterize, and automate the create-cluster → run-jobs → delete-cluster pattern for ephemeral clusters.

2 · Check your understanding