question

Sithu-5328 avatar image
0 Votes"
Sithu-5328 asked vipulsparsh-MSFT answered

Azure Sentinal - how to fetch large result set of Winsec events by pagination

Hi Community,

We pump the logs of Window security events of some computers into Azure Sentinel SIEM. Now we retrieve those logs from Sentinel to local database by using REST API. The problem is when the result set is large, the API return error message like "Result size too large". So we want to implement pagination and fetch the data from SIEM then store it in local DB.

However, according to MS docs, Kql doesn't support "Skip" operator.
We can call API multiple times by fetching 1 hr data which may be less than the limited result size but we would like to fetch many window security Event IDs in one call if possible.

So are there any ideas how to implement this pagination method to fetch the large result set from SIEM without multiple API calls?

Thanks in advance.

microsoft-sentinel
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

vipulsparsh-MSFT avatar image
0 Votes"
vipulsparsh-MSFT answered

@Sithu-5328 The problem with more number of data with REST API brings limitations like "Result size too large" to avoid pressure on infrastructure networks across the services as they are build to a certain limit of data travel.

For your particular scenario, can you utilize Event hub to push logs from it to your local database. You can also queue those logs into event hub where your local database can pull from as a consumer of those messages. Grabbing the logs as it flows through may be a better alternative.

Check more here : https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/stream-monitoring-data-event-hubs





Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.