Log Analytics Workspace: understanding data ingestion sources

Question

I recently activated a Log Analytics Workspace for my AKS cluster and I had half of my ingested data coming from "LogManagement", while the other half from "ContainerInsight".

What is "LogManagement" and how can I reduce those ingested data?

Answer 1

@Nicholas Fiorentini Welcome to Microsoft Q&A platform and thanks for using this forum.

“LogManagement” is one of the solution with in Log Analytics. You should be seeing Alert, AzureDiagnostics, AzureActivity, AzureMetrics, Perf, Usage , Heartbeat and many more tables are listed under LogManagement solution (group by the tables by solution in the Log Analytics workspace Logs explorer) . Please refer to Azure Monitor Logs table reference documentation for additional reference. Please be noted that the documentation is still under construction , however you would certainly get references to subset of the tables.

To reduce the data ingestion, you might have to isolate which DataType and resource is ingesting more data and take appropriate actions to reduce it.
You can use the below sample queries to understand which DataType is performing more billable data ingestion with in “LogManagement” solution.

Usage   
| where TimeGenerated between(datetime("2020-08-15 00:00:01") .. datetime("2020-08-20 23:59:00"))  
| where StartTime >= startofday(ago(31d)) and EndTime < startofday(now())  
| where IsBillable == true  
| summarize BillableDataGB = sum(Quantity) by Solution, DataType  
| sort by Solution asc, DataType asc  
| where Solution == "LogManagement"  

Below is the sample query to get the resource group with billable data bytes with in “AzureDiagnostics” source.

find where TimeGenerated between(datetime("2020-08-15 00:00:01") .. datetime("2020-08-20 23:59:00")) project _ResourceId, _BilledSize, _IsBillable  
| where _IsBillable == true   
| where source_ == "AzureDiagnostics"  
| summarize BillableDataBytes = sum(_BilledSize) by _ResourceId  
| extend resourceGroup = tostring(split(_ResourceId, "/")[4] )  
| summarize BillableDataBytes = sum(BillableDataBytes) by resourceGroup | sort by BillableDataBytes nulls last  

Below is the sample query to get the resource ID with billable data bytes with in “AzureDiagnostics” source.

find where TimeGenerated between(datetime("2020-08-15 00:00:01") .. datetime("2020-08-20 23:59:00")) project _ResourceId, _BilledSize, _IsBillable  
| where _IsBillable == true   
| summarize BillableDataBytes = sum(_BilledSize) by _ResourceId, source_ | sort by BillableDataBytes nulls last  
| where source_ == "AzureDiagnostics"  

Please modify the queries as per your need, for additional reference on usage queries please refer here.

Hope this information helps, please feel free to revert back if you have any further queries. Thank you