question

MennoDolstra-3373 avatar image
1 Vote"
MennoDolstra-3373 asked LyubomirAngelov-2513 commented

Lines in Azure ContainerLogs table missing compared to Kubernetes Logs

We have an AKS cluster in westeurope with 1 linux node and 2 windows nodes. On the windows nodes there are multiple pods running. The .NET Core applications inside these pods are generating application logs (stdout) which we can read by running the "kubectl logs" command. These logs contain all the information we expect to see there. The container logs are also ingested into an Azure Log Analytics workspace by using an OMS agent which runs on the linux node.
However when we query the ContainerLog table we see a lot of the same logs compared to the output of "kubectl logs" but not everything. The missing log lines are of the same log level as the present log lines and originate in the same application. Also when we access the OMS agent by going into the container we see the logs we expect in the log file present in c:/var/log/containers.
We can't figure out why some lines from the same application are present in the ContainerLogs table and others are not.

Any help would be greatly appreciated,
Menno

azure-monitorazure-kubernetes-service
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @MennoDolstra-3373 - Thanks for reaching out and for being patient. Are you still facing these issues? Please let us know, so we can investigate this further.

0 Votes 0 ·

Hi @Monalla-MSFT

We are indeed still facing this issue, so any help would be greatly appreciated.

1 Vote 1 ·

Hello,

We are facing the same issue, have you gotten any more information on the topic?
- We see logs in pod logs
- We see the logs stored in the oms agent /var/log/containers
- We do not see some of the logs in LAW
- We use a Pay Per use subscription so we should not be hitting any specific limits, we have also tried the following queries in the LAW Logs section to check for blocked logging:
Operation | where OperationCategory == "Ingestion" | where Detail startswith "The rate of data crossed the threshold"
Operation | where OperationCategory == 'Data Collection Status'


Best Regards,
Lyubomir





0 Votes 0 ·

0 Answers