Problem with starting on of my Azure Ubuntu 18.04 server in Azure

Question

I have a problem with starting on of my Azure Ubuntu 18.04 server. Gives me "grub_calloc not found" and the entering resource mode. I do have backups, and have restored 2 already, but get the same problem. How could I fix this issue?

Answer 1

Apologies for the delay. I understand you are experiencing issues with your Ubuntu 18.04 VM after updating grub2 packages
and that Boot Diagnostics shows 'error : 'symbol 'grub_calloc' not found'

This is a known issue that is currently impacting Linux distributions using grub2

• You can read about it in the following link released by Canonical: https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/GRUB2SecureBootBypass?mkt_tok=eyJpIjoiTUROa1pEbGlZell6TjJJdyIsInQiOiI3MDJlcmdDVkVySHQ4blV6a3dsMlBJb0VKMTRSemNIOEFCZ0tnVDlcL3p1ZUVLNnNlSkozRjlCN1wvRVU5MGJxZ0dFaEdvNWRcLzBGSlF0MjNXekVkYmdpaUdxQW1LaFwvUHdyRWZ2SGZjcWtcLyttXC9xVWUwbDVmRWk5RCtJZ2IyNUU2SyJ9#Recovery

• At this moment the official recommendation from Canonical is to rollback to the previous grub2 packages version, however we tested the recovery by simply reinstall grub in your affected VM and that works too.

• In the case of azure, if you created a full backup or snapshot prior updating the grub2 packages, you can rollback to that backup

• If that's not an option, then you will need to perform the process provided by canonical to downgrade grub2 from a rescue VM or simply reinstall grub, as provided below:

• You can create a rescue VM using the azure repair VM feature. This feature creates a virtual machine that you can use to repair your current OS, it will create a copy of the current OS disk and attach it to that repair VM. Of that way you can get the OS file system mounted, chroot to it and do the recovery process (downgrade grub2 packages or reinstall grub). Once that's done, you can use the same repair VM script to destroy that temporary repair VM and swap that copy disk with your original OS disk in the impacted VM, using the repair VM restore option

https://learn.microsoft.com/en-us/azure/virtual-machines/troubleshooting/repair-linux-vm-using-azure-virtual-machine-repair-commands

• You could also use a more manual approach and get a rescue VM created manually, you can find instructions in the following link :
https://learn.microsoft.com/en-us/azure/virtual-machines/troubleshooting/troubleshoot-recovery-disks-portal-linux

• Despite of what the canonical link is mentioning, there's no need to delete the existent VM to get this fixed, you can create a copy of the OS disk, modify that copy and then use the disk swap feature to boot the VM over the modified disk using any of the approaches I mentioned above.

Below the recovery details to get the VM back without downgrading grub2 packages:

Get the rescue VM created. If you use the repair VM feature, this one will create the copy of the OS disk for you and attach it to the repair VM.
Login to the rescue VM and get the root disk mounted, chroot to it and reinstall grub.
Steps below:

$ sudo su –   
# mkdir /rescue             
# mount /dev/sdc1 /rescue              
# for fs in {proc,sys,tmp,dev}; do mount -o bind /$fs /rescue/$fs; done               
# cd /rescue  
# chroot /rescue               
# lsblk <-- this will identify the attached disk, usually /dev/sdc               
# grub-install /dev/sdc              
# exit              
# cd /              
# for fs in {proc,sys,tmp,dev}; do umount /rescue/$fs; done            
# umount /rescue  

Detach the disk and swap the OS disk of your impacted VM with this one (you can use the restore option of the repair VM feature which will do this step for you)

Here is a doc for details on chroot: https://learn.microsoft.com/en-us/azure/virtual-machines/troubleshooting/chroot-environment-linux

'Please Accept as answer if it helped, so that it can help others in the community facing the same issue'