Architecture for Log data analysis

Question

Dear Team,

We have a requirement to ingest the Network user log traffic data and build the Power BI report on top of it.

Since it’s a log data, we don’t want to store it in our Azure SQL Warehouse DB, and we would like to implement any other effective architecture in azure to ingest and store this data.

Log Data Sample Below:

Source for this file:
Network application from vendor will generate this file on hourly basis and transfer this file to our SFTP. So we can say that we need to process 24 files on each day with the huge log data in each file.

Please advise me on which architecture and which service in azure is best fit for this scenario, I prefer to use some live streaming to ingest the data. Kindly share your suggestion.
Thanks,
Divakar

Answer 1

Hey,
Assuming you do have Azure SQL DW already available,
you can leverage ADF to upload files from SFTP to Azure blob , Then leverage external table to read data from file rather than explicitly loading data into synapse.
And you can query the external table via PowerBI.

Else you can leverage ADF to upload in blob and access blob file directly via PowerBI.

Answer 2

Hi @Divakar Krishnan
plan B would be to use Azure Data Factory with Log Analytics.

1. 2M records for a log analytics workspace should be a joke.
   I work with billions of records per day using Sentinel, which depends on log analytics for everything.

If this helps please accept my solution and upvote.
Or just have a nice day.