How to do big data analysis on Append Block with HDInsight or any other alternative service on Azure? (source log data produced by Azure Monitor)

Ivy Feng 16

Hello,

I enable the Diagnostic settings in Monitoring of storage account, and the log will be sent to another storage account.
And the default type of the log JSON file is append blob rather than block blob, and seems the type can not be changed.
(I think all the log produced by Azure monitor is Append Blob?)

I am trying to use HDInsight to analyze the log.
However, since HDInsight only support block blob, and does not support append blob. Therefore, it seems I am not able to analyze the log by HDInsight.

Can I ask:

1.If it is possible still use HDInsight to do big data analyze of my log without doing download and upload my file as block blob.(Since I have huge amount of log file with quite huge size as well, so it is not possible for me to write function to download the upload all of them in order to change the blob type).
Can I have any idea on how to use HDinsight to analyze the Diagnostic log(append blob) produced by Azure Monitor?

2.If it is impossible continuing using HDInsight to do analyze, if there is any alternative way(alternative service on azure) to do big data analyze directly on append block?

Thank you in advance!

Regards,
Ivy

MartinJaffer-MSFT 26,036 Reputation points

2021-07-23T22:18:14.557+00:00

Hello @Ivy Feng and welcome to Microsoft Q&A.

Usually, Log Analytics is used to analyze Azure Monitor logs.

Is there a particular reason you want to use HDInsight?
Ivy Feng 16 Reputation points

2021-07-28T03:09:05.097+00:00

Hi Martin,

Since our log is from many different sources with huge amount, we want to do some pre-process including filter, aggregation ... , then stored to db like Kusto

Therefore we are trying to find a big data service to do these kind of thing on append block.

Regards,
Ivy