question

KartikBhat-7119 avatar image
2 Votes"
KartikBhat-7119 asked MaxTaggart-2843 published

syslog/messages/kern.log rollover not working on aks nodes

We saw that after few months the aks node had diskpressure which lead to the pods being evicted, on debugging the cause for disk pressure it was observed that the syslog/messages and kern.log together were consuming around 14G of memory which was more than 50% of the free space available on the node

 -rw-r-----   1 syslog    adm             1.2G Jan 21 16:22 kern.log.1
 -rw-r-----   1 syslog    adm             6.4G Jan 21 16:22 syslog.1
 -rw-r-----   1 syslog    adm             6.4G Jan 21 16:22 messages.1

Checking the syslog config in /etc/logrotate.d/rsyslog, I can confirm that the syslog is set to rotate daily with max backups set to 7.

If this was working there should have been 8 files in total i.e, syslog, syslog.1, syslog.2.gz .... syslog.7.gz which is not the case.

Also, I see that the logs are not being written to /var/log/syslog instead are being written to /var/log/syslog.1 which is also the case with /var/log/messages and /var/log/kern.log which might be the reason why the log rotation is not working.

 -rw-r-----   1 syslog    adm                0 Jan 21 16:27 syslog
 -rw-r-----   1 root      root               0 Jan 21 16:28 kern.log.

As you can see above the size of syslog, kern.log is 0

Here are the details about the node

 Linux aks-xxxxx 5.4.0-1026-azure #26~18.04.1-Ubuntu SMP Thu Sep 10 16:19:25 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
    
    
 Node Image Version : AKSUbuntu-1804-2021.01.06
    
 Distributor ID: Ubuntu
 Description: Ubuntu 18.04.5 LTS
 Release: 18.04
 Codename: bionic



azure-kubernetes-service
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

MaxTaggart-2843 avatar image
0 Votes"
MaxTaggart-2843 answered MaxTaggart-2843 published

I'm seeing the same problem and it is causing pods to be evicted. Here is the disk usage for the relevant files in /var/log, note that both "messages" and "syslog" are 0 bytes, but there are files called "messages.1" and "syslog.1" which are both occupying 52GB:

 12K    localmessages
 0      messages
 52G    messages.1
 1.9M    messages.2.gz
 0      syslog
 52G    syslog.1
 1.1M    syslog.2.gz
 876K    syslog.3.gz
 4.0K    sysstat


Here are the contents of "/etc/logrotate.d/rsyslog":

 /var/log/syslog
 {
     rotate 7
     daily
     missingok
     notifempty
     delaycompress
     compress
     postrotate
         /usr/lib/rsyslog/rsyslog-rotate
     endscript
 }
    
 /var/log/mail.info
 /var/log/mail.warn
 /var/log/mail.err
 /var/log/mail.log
 /var/log/daemon.log
 /var/log/kern.log
 /var/log/auth.log
 /var/log/user.log
 /var/log/lpr.log
 /var/log/cron.log
 /var/log/debug
 /var/log/messages
 {
     rotate 4
     weekly
     missingok
     notifempty
     compress
     delaycompress
     sharedscripts
     postrotate
         /usr/lib/rsyslog/rsyslog-rotate
     endscript
 }

Here is the node info:

 Static hostname: aks-parallel-xxxxx-vmss000000
          Icon name: computer-vm
            Chassis: vm
         Machine ID: 3e2b3448a01e40ecaf65fc456f44a25d
            Boot ID: 5a9449cc806d4cb7a29018e26af124b7
     Virtualization: microsoft
   Operating System: Ubuntu 18.04.5 LTS
             Kernel: Linux 5.4.0-1036-azure
       Architecture: x86-64
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.