question

CannonChris-5226 avatar image
5 Votes"
CannonChris-5226 asked BrendonHolt-7726 commented

Problems with Hyper-V on Server 2019 (1809) after August 2020 patches

We've been having problems after installing August 2020 Patches on our 2019 HyperV hosts. We have multiple hyper-v clusters across Dell VRTX and UCS blades w/ ISCSI backend SANS. Both environments have seen backup times double. Additionally loading a VMs settings in HyperV or Failover cluster manager is taking a very long time. We are not using a 3rd party AV but defender managed by SCCM. Usually when i see issues like this it feels like a storage performance issue but i'm seeing the issue across the board with iscsi as well as direct attached.

Patches applied
KB4566424
KB4565349
KB4569776

our change log indicates no other changes. Our hyperv hosts have no other roles.

I'm going to try roll back the august patching and do a quick A/B test to see if that remediates the issue, but thought i'd post to see if anyone else has seen this issue.

Chris

windows-server-2019windows-server-hyper-v
· 26
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

We are also seeing delays on our 2019 Cluster. This manifests in slow "Get-VM" commands on the nodes and slowness opening settings for any of the VMs. It also seems to be causing delayed stats coming into Veeam One - sometimes these can be 15 minutes behind.

I don't like the idea of uninstalling updates to fix the issue - essentially if Microsoft is aware of the issue, they need to fix it! It could also affect anyone that has ISO compliance requiring fixes to be applied within a particular time window.

Does anyone have a link or reference to the Hotfix (Hotfix ID?) which solves the issue? I don't mind opening a case with Microsoft, if I need to go down that route to receive the fix, but it will be quicker if I can point them in the right direction.

2 Votes 2 ·

I don't think there is a public hotfix yet, only a private hotfix created ad hoc. You'll have to wait for a general release.
Alex

1 Vote 1 ·

I'm not sure i feel comfortable sharing the hotfix MS provided me as i'm not sure it was anywhere close to a finished product. I should add that I am not comfortable putting this on any of our production 2019 Hyper V servers.; I'm waiting on the general release.

1 Vote 1 ·
Show more comments

Add us to the mix, this weekend (Thanksgiving) we decided to update the Hardware Platform. Installation of Server 2019 went fine and updates were applied, up until November.

We experienced VERY SLOW Hyper-V Manager to Create or to Manage Settings on the Two New Super Powerful Computers just installed. Testing showed that during Hyper-V Manager could slow Guests down so much login to RDS servers was slow/failed. Was just taking forever to work in settings. Sunday we re-installed 2019 Server on one Machine and it was fine until updates were applied.

We decided to apply updates and deal with this issue because we provide this platform to end users, and the Legal/Financial Liability from Zero Net Logon outweighs the speed of IT Management. NOTE WE CANNOT WORK IN HYPER-V MANAGER IN PRODUCTION HOURS, 11PM to 2AM.

This BUG NEEDS TO BE FIXED and it is a SEVERE LIMITATION. I understand we all work in a complex and dangerous world, but hopefully we get a Fix SOON for this issue.

2 Votes 2 ·

Hi,

Yes, you can roll back to the previous version to see if the question still exists. And you can update your result here.

And if anyone has the same issue, welcome to post your question here.

Best Regards,
Daniel

1 Vote 1 ·

I removed only KB4565349 from a 5 node cluster (150 vms) on UCS blade chasssis and from a smaller 2 node cluster (dell vrtx) with around a dozen VMs. With the larger cluster, i was previously waiting on the window to open for a VM settings from FCM was taking between 20-40 seconds. I'm loading them now between 3 - 5. I do not yet have statistics on backups; I had to pause backups on the larger cluster b/c the job was running past the maintenance window. I'm re-enabling that back up now. With the smaller cluster, i made the change after backups had completed so i don't have a comparison.

With it being a longer holiday weekend i'll likely let things sit as they are over the weekend. If backups seem back to normal, i'm open to do some more testing with a couple other clusters (brand new UCS blades). To my knowledge, we're not seeing any VM performance issues at this point.

Thanks Chris







1 Vote 1 ·

Sounds good.

Please let me know if you have any other questions.


1 Vote 1 ·

This is certainly happening on clusters we run. We have not yet rolled back (more complicated than removing a single update) and inherently puts our environment in risk by running on an old rollup if we do.

Very frustrating to not have Microsoft already patched this, so we'll likely burn a support case to get a hotfix.

Can you provide any estimate on resolution or acknowledgment that Microsoft has even identified this?

1 Vote 1 ·

I just heard back. The update is scheduled for inclusion in January.

3 Votes 3 ·
Show more comments

Its been a month since i've heard from them. I pinged them a moment ago to see if there is a schedule for it to be included in the normal rollup. Believe me... I share your frustration. It took several calls/screen shares with multiple teams to get anyone to really acknowledge the issue. I'm still waiting on it to be rolled into the monthly cumulative. I'll respond if i hear back.

Chris

1 Vote 1 ·
Show more comments

On a 6 Nodes Win2019 SCVMM managed FailOverCluster, that we use for Automated Testing , which heavily relies on CheckPoints we have these findings:

  • Win2019/SCVMM2019 has double duration, when operations are performed on Non SCV owner, whereas Win2016/SCVMM2016 has no measureable difference.

  • Win2019+KB4586793 doubles Hyper-V Restore on Non owner. SCVMM Restore is four times slowere, mostly affected by prolonged Refresh durations

  • Win2019+KB4598230 adds 30% to Hyper-V Restore. Doubles SCVMM Restore duration,

  • Eventually MigrateVm AWAY from SCV Owner ends up lasting 6 – 14 minutes

Measurements when Restoring a complex CheckPoint (Numbers are seconds):

56592-image.png

56439-image.png

56440-image.png

So January Update was even worse!

0 Votes 0 ·
image.png (5.4 KiB)
image.png (5.6 KiB)
image.png (5.7 KiB)

BTW: Due to lots of SCVMM Error ID 2606 From the superslow Refresh(*), we have added 'Retry' SCVMM Command' for ERRORID:2606 to our Test Automation Execution , but are still severely down on Test Thrioughput.. We are still on KB4586793!


Unable to acquire a 'Delete' lock on object '52112b4e-3c4c-42b0-a2d9-0e708231d1c5' of type 'VirtualHardDisk' because it is locked by task 'ec55c407-f8dc-4900-9f3a-ce570d4a5d02' 'Refresh host cluster' with a 'Write' lock. (Error ID: 2606)

0 Votes 0 ·

For completeness, we changed VmDatStore.dll back to the latest available in WinSxS before August 2020, an got these numbers - so no relief compared to August, but sime relief compared to the Jan 2021 results (We have only tried this on a Staging environment!)

.56671-image.png


0 Votes 0 ·
image.png (19.7 KiB)

Hello Jens.
Have you been in talk with Microsoft regarding this issue? :)

0 Votes 0 ·
Show more comments

We opened a ticket with MS and referenced this post and the case# CannonChris-5226 provided. We were provided a private hotfix that is time limited which they stated they have had for a few months. It was time stamped in October 2020. Installing this hotfix requires using bcdedit to put and leave the server in test mode until the patch is removed and replaced with the public version. They also make accept a bunch of warnings that the hotfix is provided as-in and may cause other issues.

Given the impact on our environment and that there were no issues encountered in our test environment we have put this into production and it immediately resolved the IO performance issue. It did not solve the issue with VM settings taking a long time to load or the right click menu in cluster manager disappearing when you try to select something.

0 Votes 0 ·
Show more comments
OneNerdyGuy-8308 avatar image
0 Votes"
OneNerdyGuy-8308 answered OneNerdyGuy-8308 published

how does one go about replacing the VmDataStore.dll? I've got a copy from an unpatched I had laying about, but everything I've tried gives me issues. Do I have to go do it via Safemode?

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

CannonChris-5226 avatar image
0 Votes"
CannonChris-5226 answered CannonChris-5226 commented

I'm going off a memory that is nearly 6 months ago, but you'll need to stop hyperv services. I renamed the current one .dll08 (also think i put it in another CYA type of folder. You'll probably have to take ownership of the file to rename it. Then copied the new one in. Gave the server a reboot. I'm not sure it was super complicated but its been months now.

Just so you know, I reached out to all my MS reps to get more information and one was very useful in reaching out to the tech that worked my case. The word I got back that the release of the hotfix into the the rollup was supposed to go out last month but there were adjacent issues that were ideally included but this is taking place in the March 2021 Rollup. The 2nd Tuesday of march comes fairly early (9th); If it were me i'd hold off. I wasn't holding my breath before but I Trust this person so i am very hopeful and willing to wait another 11 days.

CC

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Yeah, this was more of a 'if March didn't happen", but I appreciate the insight. This has been a real pain haha!

0 Votes 0 ·

Just to confirm, have you seen a confirmation that the fix is included in today's patches?

0 Votes 0 ·

I haven't received anything from support but up to this point i've had to reach out them. We're building SCCM update deployments this afternoon. I"ll likely deploy the patches tonight and see. Fingers are crossed.

0 Votes 0 ·
PaulWebb-5946 avatar image
1 Vote"
PaulWebb-5946 answered CannonChris-5226 commented

Looks like we are out of luck. I don't see anything but a Microcode update. Does anyone still have a support channel open with Microsoft to find out if they ever plan to fix?

· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

What version of the DLL are you seeing on your updated hypervisors?

0 Votes 0 ·

Just installed Hyper-V Core 2019 on a machine and updated to the latest patches including KB5000822.

Here I got version 10.0.17763.1697 of the VMDataStore.dll, signed at 2021-01-07.

0 Votes 0 ·

Do you see the performance impact like before, or is it resolved?

0 Votes 0 ·
Show more comments
PaulWebb-5946 avatar image
0 Votes"
PaulWebb-5946 answered

Same ....10.0.17763.1697 of the VMDataStore.dll

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PaulWebb-5946 avatar image
0 Votes"
PaulWebb-5946 answered PaulWebb-5946 commented

I have installed the latest updates over the weekend and Veeam backup performance has dropped through the floor. We are also seeing....

.....14/03/2021 01:38:40 :: Unable to allocate processing resources. Error: Unable to find Hyper-V hosts where VM 'f81942a3-3233-430a-bf38-7da709ad7a72' is registered: Failed to collect all VMs. Error code: 0x80041032
Cannot query class instance from enumerator object
[WMI] Failed get next object in collection.

What is everybody seeing?

· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

We did not apply the update, without a confirmed case of it fixing the issue. Sorta didn't want to make it worse.

0 Votes 0 ·

Yeah get that. We were getting a bit desperate and were hoping for the best.
Putting a case together to move to VMWare - completely lost confidence in Microsoft to fix anything!

0 Votes 0 ·

We applied updates last week and initially it looked good. Cluster manager was very responsive again, right click menu worked and Veeam backup performance improved but was not fantastic. Since then, with no changes all issues and poor performance have returned. Has anyone contacted MS support again so see what their response is? Wonder if they actually released the patch. I have no time to call them this week but will try next week if nobody else has yet.

0 Votes 0 ·
Show more comments
TobiasLehner-6747 avatar image
0 Votes"
TobiasLehner-6747 answered SteenPedersen-9106 commented

did anyone try KB5000854 if it fixes our issue?

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

The DLL is brand new... there's hope
82398-image.png


0 Votes 0 ·
image.png (31.8 KiB)

I have just installed it on a few servers and it works! Really nice to have the hyper-v manager respond within seconds again.

0 Votes 0 ·
JoshHargense-1009 avatar image
0 Votes"
JoshHargense-1009 answered SteenPedersen-9106 commented

OMG, it is so amazing to have hyper-v manager (and SCVMM) responding in a timely manner!

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Did the install require a reboot?

0 Votes 0 ·

Yes it does require a reboot.

0 Votes 0 ·
CannonChris-5226 avatar image
0 Votes"
CannonChris-5226 answered

all the chatter got me to sign on and do some deployments to Dev/Test and our remote sites. Can't wait to see. I'm sure MS is tired of hearing from me as much as i'm tired of reaching out with a WTF message

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

CannonChris-5226 avatar image
0 Votes"
CannonChris-5226 answered

all the chatter got me to sign on and do some deployments to Dev/Test and our remote sites. Can't wait to see. I'm sure MS is tired of hearing from me as much as i'm tired of reaching out with a WTF message

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

CannonChris-5226 avatar image
0 Votes"
CannonChris-5226 answered

all the chatter got me to sign on and do some deployments to Dev/Test and our remote sites. Can't wait to see. I'm sure MS is tired of hearing from me as much as i'm tired of reaching out with a WTF message

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.