Watching a transformation job's health and progress

Monitoring data transformation in Microsoft Fabric involves tracking the execution, performance, and status of Dataflows Gen2, pipeline activities, and Spark notebook/job runs to ensure reliability and troubleshoot failures. Fabric provides centralized visibility through the Monitoring hub, item-level run history, and Spark application details, enabling engineers to identify bottlenecks, errors, and resource consumption.

1 · Learn the must-know

• 1The Monitoring hub in Fabric provides a unified view across the workspace showing run status, duration, and trigger type for pipelines, dataflows, and notebooks.
• 2Each Dataflow Gen2 run can be inspected individually to see refresh history, duration, and step-level errors, which helps isolate transformation failures.
• 3Pipeline activity runs expose detailed input/output JSON, error messages, and execution duration per activity, which is essential for debugging Copy or Dataflow activities embedded in pipelines.
• 4Spark job monitoring for notebooks and Spark job definitions surfaces the Spark application UI, including stages, tasks, executors, and logs for diagnosing performance issues or failures in transformation code.
• 5Fabric retains run history for a limited time (subject to workspace/capacity settings), so long-term monitoring or auditing typically requires exporting logs or integrating with Azure Monitor/Log Analytics.
• 6Failures in scheduled dataflow or pipeline runs can be configured to send notifications (e.g., via Fabric alerts or Power Automate/Outlook integration), so learners should know how to set up proactive alerting rather than relying solely on manual checks.

2 · Check your understanding