Spotting performance trends in a job's run history

The Lakeflow Jobs run history view lists every past run of a job with start time, duration, trigger type, and status, letting engineers spot whether a job is slowing down, speeding up, or failing more often over time. Comparing recent runs against a historical baseline (e.g. typical duration over past weeks) helps distinguish a one-off anomaly from a genuine performance regression. This view is the primary UI-based tool for trend analysis before diving into Spark UI or cluster metrics for root cause.

1 · Learn the must-know

• 1Run history is accessed from the Jobs UI under a specific job's 'Runs' tab, showing a chronological list with duration, start/end time, trigger (scheduled, manual, API), and status (succeeded, failed, running).
• 2Sorting or scanning run duration over time reveals trends such as gradually increasing runtime, which often points to data volume growth, skew, or cluster under-provisioning rather than a sudden failure.
• 3Each run's detail page links to the Spark UI for that run, enabling drill-down from a duration anomaly into stage/task-level bottlenecks.
• 4Comparing a current run against historical baselines helps separate transient issues (e.g. cloud provider throttling, spot instance eviction) from systemic problems (e.g. an unoptimized query or schema drift causing shuffle-heavy operations).
• 5Job clusters that use autoscaling or new cluster-per-run configurations can introduce duration variance unrelated to code changes, so baseline comparisons should account for cluster startup/provisioning time separately from task execution time.
• 6Repeated runs with increasing duration despite stable data volume is a common gotcha suggesting cluster resource contention, small file problems, or the need for table optimization (e.g. OPTIMIZE/Z-ORDER on Delta tables).

2 · Check your understanding