Setting up a pipeline that loads new data on its own as files arrive

Automated data ingestion in Snowflake is primarily achieved through Snowpipe for continuous file-based loading and Snowpipe Streaming for low-latency row-based ingestion, both eliminating the need for manual COPY INTO execution. These services integrate with cloud storage event notifications or REST API calls to trigger loads automatically as new data arrives.

1 · Learn the must-know

• 1Snowpipe uses COPY INTO statements defined within a pipe object and loads data automatically when new files land in a stage, triggered either by cloud provider event notifications (SQS/SNS, Event Grid, Pub/Sub) or REST API calls.
• 2Snowpipe is designed for micro-batch loading of files and incurs charges based on compute resources used per file (with per-file overhead), making it less efficient for very small or extremely frequent files compared to Snowpipe Streaming.
• 3Snowpipe Streaming ingests rows directly from a client application (via the Snowflake Ingest SDK) without staging files first, reducing latency and cost for high-frequency, low-latency use cases like Kafka-based pipelines.
• 4The Kafka connector for Snowflake leverages Snowpipe or Snowpipe Streaming under the hood to automatically ingest data from Kafka topics into Snowflake tables.
• 5Automated ingestion via Snowpipe does not guarantee exactly-once delivery in all edge cases (e.g., duplicate notifications), so idempotent design or deduplication logic may be needed; Snowflake also does not guarantee the order of file processing.
• 6Pipes are paused automatically if the underlying stage or file format is dropped/altered incompatibly, and monitoring should be done via SYSTEM$PIPE_STATUS, PIPE_USAGE_HISTORY, or COPY_HISTORY views rather than assuming ingestion always succeeds silently.

2 · Check your understanding