Applying retention rules to data

Data retention in Unity Catalog is managed at the Delta table level through table properties that control how long deleted files and transaction log history are kept, enabling Time Travel while balancing storage costs. Data engineers must understand the interaction between VACUUM, retention properties, and Unity Catalog governance to safely purge or preserve historical data.

1 · Learn the must-know

• 1The Delta table property delta.deletedFileRetentionDuration (default 7 days) controls how long tombstoned data files are retained for Time Travel queries before VACUUM can remove them.
• 2The Delta table property delta.logRetentionDuration (default 30 days) controls how long transaction log JSON/checkpoint files are kept, limiting how far back Time Travel can go.
• 3Running VACUUM with a retention period shorter than the default 7 days requires disabling the safety check via spark.databricks.delta.retentionDurationCheck.enabled = false, and is discouraged because it can break concurrent readers or Time Travel queries.
• 4VACUUM only removes files no longer referenced by the current table version plus files older than the retention threshold; it does not delete data still needed by active transactions.
• 5Table properties for retention are set or altered using ALTER TABLE ... SET TBLPROPERTIES and apply per-table, so governance/retention policies must be applied consistently across managed and external Unity Catalog tables.
• 6Unity Catalog audit logs and system tables (e.g., system.access.audit) have their own retention behavior separate from Delta table data retention, and are used for compliance/governance tracking rather than data recovery.

2 · Check your understanding