The update policy documentation for Azure Data Explorer makes the following statements about transactional update policies (these statements are inconsistent with each other): "guarantees consistency between data in the source and target table" "depending on conditions, sometimes data is ingested to the source table, but not to the target table" Which is true? Does a transactional update policy guarantee that if target table ingestion fails, then source table ingestion fails also (atomic operation). If not then it is not transactional and this miscommunication needs amending. Clarity would be much appreciated. Thanks in advance.

ADX Transactional Update Policy Guarantees

Alex Lorimer 0

The update policy documentation for Azure Data Explorer makes the following statements about transactional update policies (these statements are inconsistent with each other):

"guarantees consistency between data in the source and target table"
"depending on conditions, sometimes data is ingested to the source table, but not to the target table"

Which is true? Does a transactional update policy guarantee that if target table ingestion fails, then source table ingestion fails also (atomic operation). If not then it is not transactional and this miscommunication needs amending.

Clarity would be much appreciated. Thanks in advance.

PRADEEPCHEEKATLA-MSFT 77,901 Reputation points Microsoft Employee

2024-03-11T08:51:30.21+00:00

@Alex Lorimer - Thanks for the question and using MS Q&A platform.

The statements you mentioned from the Azure Data Explorer update policy documentation seem to be inconsistent with each other.

Based on what you have mentioned, it seems that setting IsTransactional:true in the update policy does guarantee consistency between data in the source and target table. However, if the policy conditions fail, data will not be ingested to the source table.

In some cases, data may be ingested to the source table but not to the target table, depending on the conditions specified in the policy. This means that the update operation may not be atomic, as the ingestion to the target table may fail while the ingestion to the source table succeeds.

Therefore, it is important to carefully consider the conditions specified in the update policy to ensure that the desired consistency and atomicity guarantees are met.

Hope this helps.
Alex Lorimer 0 Reputation points

2024-03-11T09:40:09.11+00:00

Could you elaborate on "In some cases" (when would data ever get committed to the source table while failing to write to the target table). You're clearly describing a non-transactional operation, and therefore IsTransactional:true is extremely misleading to the point of being wrong... My expectation is that the documentation is inaccurate (it's currently inconsistent with itself).

Is your answer based on having had a word with the dev team, or are you just basing your answer on the inconsistent documentation?
PRADEEPCHEEKATLA-MSFT 77,901 Reputation points Microsoft Employee

2024-03-11T10:40:04.88+00:00

@Alex Lorimer - Yes, I agree that the document needs to be fix this issue.

Could you please share the feedback on the document to fix this issue as shown below?

Hope this helps.
Alex Lorimer 0 Reputation points

2024-03-11T11:26:15.71+00:00

Thanks, I've done that now. Upon submitting the feedback I received a notice that direct responses to feedback aren't given. Where should I await a response that clarifies the transactional guarantees provided by ADX update polices?

Is this a better place to move the conversation? -> https://feedback.azure.com/d365community/idea/d5752e4c-99df-ee11-a73d-000d3adc65a4
PRADEEPCHEEKATLA-MSFT 77,901 Reputation points Microsoft Employee

2024-03-12T04:57:04.2966667+00:00

@Alex Lorimer - Feedback forum is suited for Product feebacks and not document feedbacks.

We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.
Alex Lorimer 0 Reputation points

2024-03-26T09:24:26.7966667+00:00

Hi Pradeepcheekatla,

Any word from the dev team yet?
PRADEEPCHEEKATLA-MSFT 77,901 Reputation points Microsoft Employee

2024-03-27T08:10:49.24+00:00

@Alex Lorimer - I'm still awaiting for the response and will get back to you as soon as we have an update.