Watching tasks, streams, and Snowpipe for health and data quality

Monitoring continuous pipelines in Snowflake means checking the health and progress of Snowpipe, Streams, Tasks, and Dynamic Tables using built-in views, table functions, and system functions rather than external tooling. Snowflake exposes both real-time status calls and historical Account Usage/Information Schema views so engineers can detect lag, failures, and stale data before they impact downstream consumers.

1 · Learn the must-know

• 1Use SYSTEM$PIPE_STATUS('pipe_name') for real-time Snowpipe status (pendingFileCount, lastIngestedTimestamp, executionState) and COPY_HISTORY (table function or ACCOUNT_USAGE view) for detailed load history and errors.
• 2PIPE_USAGE_HISTORY (Account Usage) tracks Snowpipe credit consumption and bytes/files ingested over time for cost and throughput monitoring.
• 3Task execution is monitored via the TASK_HISTORY table function/view (state, error_code, error_message, next_scheduled_time), and SERVERLESS_TASK_HISTORY tracks credits used by serverless tasks separately from warehouse-based ones.
• 4For Streams, SHOW STREAMS exposes a STALE column and stale_after timestamp, and SYSTEM$STREAM_HAS_DATA(stream_name) checks whether change data exists before consuming it, since an expired stream (past its retention/staleness window) must be recreated and loses change data.
• 5Dynamic Tables expose refresh status and lag via DYNAMIC_TABLE_REFRESH_HISTORY and SHOW DYNAMIC TABLES/INFORMATION_SCHEMA (data_timestamp, target_lag, actual latency), letting you confirm whether the configured target lag is being met.
• 6Notification integrations can be attached to pipes and tasks (error notifications) so failures or backlog conditions trigger cloud messaging (e.g., SNS/Event Grid/Pub-Sub) instead of relying solely on manual polling.

2 · Check your understanding