question

IvyFeng-8510 avatar image
1 Vote"
IvyFeng-8510 asked Sumarigo-MSFT edited

How to do big data analysis on Append Block with HDInsight or any other alternative service on Azure? (source log data produced by Azure Monitor)

Hello,

I enable the Diagnostic settings in Monitoring of storage account, and the log will be sent to another storage account.
And the default type of the log JSON file is append blob rather than block blob, and seems the type can not be changed.
(I think all the log produced by Azure monitor is Append Blob?)

I am trying to use HDInsight to analyze the log.
However, since HDInsight only support block blob, and does not support append blob. Therefore, it seems I am not able to analyze the log by HDInsight.


Can I ask:

1.If it is possible still use HDInsight to do big data analyze of my log without doing download and upload my file as block blob.(Since I have huge amount of log file with quite huge size as well, so it is not possible for me to write function to download the upload all of them in order to change the blob type).
Can I have any idea on how to use HDinsight to analyze the Diagnostic log(append blob) produced by Azure Monitor?

2.If it is impossible continuing using HDInsight to do analyze, if there is any alternative way(alternative service on azure) to do big data analyze directly on append block?

Thank you in advance!

Regards,
Ivy

azure-monitorazure-databricksazure-hdinsight
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @IvyFeng-8510 and welcome to Microsoft Q&A.

Usually, Log Analytics is used to analyze Azure Monitor logs.

Is there a particular reason you want to use HDInsight?


0 Votes 0 ·
IvyFeng-8510 avatar image IvyFeng-8510 MartinJaffer-MSFT ·

Hi Martin,

Since our log is from many different sources with huge amount, we want to do some pre-process including filter, aggregation ... , then stored to db like Kusto

Therefore we are trying to find a big data service to do these kind of thing on append block.

Regards,
Ivy

0 Votes 0 ·

0 Answers