.NET runtime errors IIS

Michael Gerholdt 26 Reputation points
2021-10-14T14:00:09.713+00:00

Seemingly out of the blue, roughly 10 am on Wednesday 10/13/2021, multiple web servers began experiencing the same issues, recurring, causing application pools to stop and disrupting web service, of course.

The last patches that were applied were on 9/17/2021, and these issues were not noted until 10/13; sudden and persistent. Restarting app pools, IISReset, server reboots ... any relief is temporary.

Since this is happening on at least 6 web servers at our location, all starting at same timeframe, it seemed likely that it was a wide-spread issue, but not seeing discussion out there. Are we alone in this? :)

Any guidance appreciated.

Application: w3wp.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an internal error in the .NET Runtime at IP 00007FF89EF3C903 (00007FF89EEB0000) with exit code 80131506.

and

Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x5215df96
Faulting module name: clr.dll, version: 4.7.3850.0, time stamp: 0x60d67f8a
Exception code: 0xc0000005
Fault offset: 0x000000000008c903
Faulting process id: 0x2d6c
Faulting application start time: 0x01d7c062446e1787
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Report Id: 824cb449-2c55-11ec-80dc-005056b00f54
Faulting package full name:
Faulting package-relative application ID:

Internet Information Services
Accepted answer
  1. Paul Solovyovsky 81 Reputation points
    2021-10-14T15:02:47.583+00:00

    For at least one customer it appears to be Cylance AV related. Putting Cylance into monitor mode appears to provide relief for the time being, we are investigating and monitoring situation.

    1 person found this answer helpful.

9 additional answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Dennis, John (NIH/NEI) [E] 11 Reputation points
    2021-10-14T19:33:28.903+00:00

    Cylance is the problem. They released an update silently on Tuesday afternoon/evening. If you are running Cylance version 2.1.1584.x. please be aware you can expect problems with IIS, Citrix ICA, and other issues like service or user account lock outs. We have had to roll back to version 2.1.1574 so critical applications and web services can function. I am waiting for a callback from Cylance as this is unacceptable. We have been in various states of down for 36hrs. If someone from Cylance is listening, Microsoft can offer a similar AI/ML AV model and is ready to talk.

    2 people found this answer helpful.
  2. Michael Gerholdt 26 Reputation points
    2021-10-14T17:30:39.94+00:00

    We put Cylance in monitor mode and that relieved the issue for us. We also noted some desktop apps which were being blocked which should not have been, also by Cylance.

    Silence Cylance and all is good.

    We're putting in a ticket with Cylance support regarding this, as suggested.

    1 person found this answer helpful.
  3. Erik S 1 Reputation point
    2021-10-14T14:36:58.587+00:00

    Started having the exact same issue. Our event viewer logs show the identical. This seems to have very little the ways of common denominators. Only that our issues seemed to begin at roughly the same time as mentioned by the OP here.

    This issue is affecting only some of our applications and even applications that share the same pool are mixed and matched as to which ones function.

    0 comments
  4. Paul Solovyovsky 81 Reputation points
    2021-10-14T14:39:40.523+00:00

    We are seeing this across multiple customers, it does not appear to be a random event. Initial issue reported between 10am-11am on 10/13/2021