We saw that after few months the aks node had diskpressure which lead to the pods being evicted, on debugging the cause for disk pressure it was observed that the syslog/messages and kern.log together were consuming around 14G of memory which was more than 50% of the free space available on the node
-rw-r----- 1 syslog adm 1.2G Jan 21 16:22 kern.log.1
-rw-r----- 1 syslog adm 6.4G Jan 21 16:22 syslog.1
-rw-r----- 1 syslog adm 6.4G Jan 21 16:22 messages.1
Checking the syslog config in /etc/logrotate.d/rsyslog, I can confirm that the syslog is set to rotate daily with max backups set to 7.
If this was working there should have been 8 files in total i.e, syslog, syslog.1, syslog.2.gz .... syslog.7.gz which is not the case.
Also, I see that the logs are not being written to /var/log/syslog instead are being written to /var/log/syslog.1 which is also the case with /var/log/messages and /var/log/kern.log which might be the reason why the log rotation is not working.
-rw-r----- 1 syslog adm 0 Jan 21 16:27 syslog
-rw-r----- 1 root root 0 Jan 21 16:28 kern.log.
As you can see above the size of syslog, kern.log is 0
Here are the details about the node
Linux aks-xxxxx 5.4.0-1026-azure #26~18.04.1-Ubuntu SMP Thu Sep 10 16:19:25 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Node Image Version : AKSUbuntu-1804-2021.01.06
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic