question

KanhaiArora-9745 avatar image
1 Vote"
KanhaiArora-9745 asked ChrisPollock-3697 answered

Logs are missing in the Log Analytics Workspace - Azure Data Factory

We came across a situation where certain logs are missing of in the ADFPipelineRun in the log analytics workspace.

For instance a log of Succeeded pipeline execution is missing even if the pipeline has finished the execution successfully. and there are several such

azure-data-factory
· 9
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @KanhaiArora-9745,

Welcome to Microsoft Q&A Platform.

Sorry that you are experiencing this issue. There is a known limitation of 500 column limit in Azure monitor logs and AzureDiagnostics table is in particular susceptible to this limit since it includes properties for all Azure services writing to it. Once this limit is reached, any rows containing data with any property outside of the first 500 will be dropped at ingestion time. Possible workarounds are given in the below document as well. Please let us know if your scenario is different and also more details - Pipeline Run ID that is missing and region of the data factory for us to investigate further.

column-limit-in-azurediagnostics

0 Votes 0 ·

Lets take example of Run id : 35f19e81-92b7-433e-86ea-5be4578472f8
Its seen in progress in Azure Logs and never succeeded.
Similar is the case with 1924241c-4665-448f-83a3-3470ea1dfe4f and many more, however there are many where all 3 entries are logged i.e. Queued, In Progress and Succeeded21898-1.jpg21878-2.jpg




We rely on these status to send out alerts if a Job is In Progress for more than 15 minutes and a missing Success Entry for actually a successful run is generating false alerts

0 Votes 0 ·
1.jpg (54.9 KiB)
2.jpg (47.3 KiB)

Hi @KanhaiArora-9745,

This is strange. We are working closely with the internal teams to understand the root cause of this. I will get back to you as soon as I hear update from them.

Thanks for your patience!

0 Votes 0 ·
Show more comments
HarithaMaddi-MSFT avatar image
0 Votes"
HarithaMaddi-MSFT answered KanhaiArora-9745 commented

Hi @KanhaiArora-9745,

Thanks for your patience.

Our internal team investigated and observed that this was caused due to outage in WestUS region where certain logs were accidentally deleted and it was confirmed that the issue is fixed now.

Please let us know if issue reoccurs and we will further investigate it with our team.

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @KanhaiArora-9745,

We have not received a response from you. Please let us know if you are still facing the issue, we will continue to engage with you.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

0 Votes 0 ·

Hi @KanhaiArora-9745,

We still have not heard back from you. Are you still facing the issue? If you found a solution, would you please share it here with the community? Otherwise, let us know and we will continue to engage with you on the issue.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

0 Votes 0 ·

I am sorry for replying late. Let me monitor this for a while and if this happens again i'll report it back

0 Votes 0 ·
MiguelKulisicPecar-4967 avatar image
0 Votes"
MiguelKulisicPecar-4967 answered

Hello,
I can confirm this issue has not been fixed:
39172-image.png



I've got multiple runs where I'm missing logs for "Queued" and "InProgress" status.


image.png (60.7 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

ChrisPollock-3697 avatar image
0 Votes"
ChrisPollock-3697 answered

I am seeing multiple issues where the Succeed records are not showing in log analytics, but the InProgress and Queued records are showing. I can actually find the ActivityRunId on the pipeline in the ADF UI and it shows as Succeeded, but there is no Succeeded record in log analytics.

This is important as we're looking at getting detailed costs down to the activity and pipeline level, but with missing records it will never be completely accurate.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.