question

slay3r9903-0375 avatar image
0 Votes"
slay3r9903-0375 asked slay3r9903-0375 commented

Linux VM on Hyper-V - Disk IO

I have several Linux VMs running on:
WindowsEditionId : ServerDatacenter
WindowsProductName : Windows Server 2019 Datacenter
WindowsVersion : 1809
CsModel : ProLiant DL380p Gen8
CsNumberOfLogicalProcessors : 48
CsProcessors : {Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz, Intel(R) Xeon(R) CPU E5-2695 v2 @
2.40GHz}
CsTotalPhysicalMemory : 360740823040
DeviceGuardAvailableSecurityProperties : {BaseVirtualizationSupport, DMAProtection}
RAID: HP P420i Controller

Running RAID 10

After a while, my Linux VMs run into an issue where they have several repeating instances of writing to the system journal. It seems like it is due to an IO issue.
Here are the VM disk settings:
Disk /dev/sda: 40 GiB, 42949672960 bytes, 83886080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt

Are there any special settings I need to set on the hyperivsor or in the VM?

windows-server-2019windows-server-hyper-v
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi,

I would like to check if the reply could be of help? If yes, please help accept answer, so that others meet a similar issue can find useful information quickly. If you have any other concerns or questions, please feel free to feedback.

Best Regards,
Joan

0 Votes 0 ·
jiayaozhu-MSFT avatar image
0 Votes"
jiayaozhu-MSFT answered slay3r9903-0375 commented

Hi,

Based on your screenshot, please first try to troubleshoot your issue with the guidance in the article below and as this article suggests, try to update your firmware of your SSD or your kernel:

https://askubuntu.com/questions/1173738/crash-systemd-journal-failed-to-write-entry-ignoring-read-only-file-system-on

Thanks for your support!

BR,
Joan


If the Answer is helpful, please click "Accept Answer" and upvote it.

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

It does look like there were some updates done in later kernels ( I was running 5.4 released 11/2019). I updated the kernels on the VMs and will see if that helps.

0 Votes 0 ·

For now, upgrading the kernel to 5.8 resolved the issue. I'll keep an eye on the VMs to ensure it doesn't happen again.
Thanks for the assist, this has been driving me nuts for months!

0 Votes 0 ·
jiayaozhu-MSFT avatar image
0 Votes"
jiayaozhu-MSFT answered slay3r9903-0375 edited

Hi,

Thanks for posting on our forum!

Based on your descriptions, firstly, I suppose, the reason why repeating instances were wirtten to your system journal was because you ran RAID10 which would automatically backup your data into your mirrored disk (i.e. each instance will be duplicated when being stored). Here is an article which can help you better understand RAID10:

https://www.enterprisestorageforum.com/software/what-is-raid-10-how-raid-10-works/

(Please note: Information posted in the given link is hosted by a third party. Microsoft does not guarantee the accuracy and effectiveness of information.)

In addition, after my research on Linux (don't know which version your Linux is), the IO setting is in default and if you do not have any other requirements for your IO setting, just let its default setting alone. Or, you can contact your Linux supporter for more details about your situation (should I increase IO if running RAID10?) But, also in my research, running RAID10 is recommended by Linux, so its default setting for IO should already take this situation into consideration.

Here is an article about default maximum IO size in Linux kernel:

https://longwhiteclouds.com/2016/03/06/default-io-size-change-in-linux-kernel/

(Please note: Information posted in the given link is hosted by a third party. Microsoft does not guarantee the accuracy and effectiveness of information.)

Thanks for your support! If you have any new findings about this case, you are encouraged to share your ideas and findings here. Have a nice day! : )

BR,
Joan


If the Answer is helpful, please click "Accept Answer" and upvote it.

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

99990-linuxvm.png


Here is a screenshot of the message I get after about a day. This only happens on Linux (Ubuntu) virtual machines, the Windows servers I have running do not throw any IO errors.
Also, I forgot that I moved these to a RAID 1 array of NVMe SSDs, and the issue persists. I'm not sure if it is an issue with the current Hyper-V setup or something else.
@jiayaozhu-MSFT

0 Votes 0 ·
linuxvm.png (78.5 KiB)