Functional Team - A Functional Company - Cloud & Data Engineering on Microsoft AzureClarify Kusto delete commands on ADX
ADX

Clarify Kusto delete commands on ADX

Christoph Thale|
#Azure#ADX#KQL#Kusto#Delete

Azure Data Explorer (ADX) provides robust mechanisms for data management, including sophisticated data deletion strategies. Understanding the differences between hard and soft deletes, as well as the role of retention policies, is crucial for effective data lifecycle management. This article delves into these concepts, helping you navigate the intricacies of data deletion in ADX.

Hard Deletes vs. Soft Deletes

In ADX, data deletion can be broadly categorized into two types: hard deletes and soft deletes. Each method serves different purposes and has distinct implications.

The key difference between hard and soft deletes is the 14-day recovery window provided by soft deletes. This period allows for potential recovery of data if needed, offering a safety net for accidental or premature deletions. If you accidentally soft-delete data, you should immediately contact the Azure Microsoft Support Team. The engineers can restore the data within the 14-day window, as previously mentioned.

Retention Policies

In addition to hard and soft deletes, ADX uses retention policies to manage data lifecycle more comprehensively. Retention policies help in defining how long data should be retained and under what conditions it can be deleted.

Retention policies in ADX are characterized by two main attributes: SoftDeletePeriod and Recoverability.

  1. SoftDeletePeriod:

    • This attribute specifies the duration for which data will be retained after the soft delete operation. For example, a policy might be configured with a SoftDeletePeriod of 3650.00:00:00 days, which equals 10 years. This means that from the time of data ingestion, the data will be kept for 10 years before it is eligible for deletion.
  2. Recoverability:

    • The Recoverability attribute determines whether soft-deleted data can be recovered after the SoftDeletePeriod has elapsed. If set to Enabled, the data will undergo a soft delete process after the retention period ends, providing an additional 14 days for potential recovery. If set to Disabled, the data will be permanently deleted from the disk once the retention period is over, without any recovery window.

Consider the following retention policy configuration:

{
  "SoftDeletePeriod": "3650.00:00:00",
  "Recoverability": "Enabled"
}

In this scenario, data is retained for 10 years from ingestion. After the 10-year mark, a soft delete is performed, allowing for an additional 14 days for recovery. If the Recoverability attribute were set to Disabled, the data would be immediately and permanently deleted after the 10 years.

That is all you have to know about deletion in my opinion but if you want to dive deeper into this topic you could read these docs:

Thank you for reading and we will see you in the next blog article.

Back to Blog