gMSA with IIS and SQL server access failes during password changes

Martin Larsen 1

We're running a series of websites configured to use gMSA as their identity.
All sites have access to our SQL server connecting with the respective gMSA account.
The SQL server have the gMSAs added to the relevant database to grant access.
Everyting is working as expected.

We're having issues when the gMSA recycles the password every month.
During recycling the website is denied SQL access, resulting in a series of failed request to the websites. It lasts for around 5 minutes.

The IIS and domain controllers are running on 2016 servers.

Is this to be expected or can we do something to resolve this?

The problem could essentially be with the Domain, IIS or the SQL server..
Any help would be appreciated.

Regards,
Martin

Bruce Zhang-MSFT 3,736 Reputation points

2022-01-18T03:19:26.397+00:00

Hi @Martin Larsen ,

During recycling the website is denied SQL access, resulting in a series of failed request to the websites.

Is there any error message when request failed? We can know which caused this error through error message, IIS or SQL Server.

If the error message points to a connection, the problem is most likely the database. Maybe the database didn't apply the recovered password in time. Otherwise it's IIS. If it's a problem with IIS, it's best to check event viewer.
Martin Larsen 1 Reputation point

2022-01-18T09:30:50.18+00:00

Thanks for your reply.

This particular website is a classic asp website, but we do get them from asp.net as well.

It seems like to error comes from the SQL server.

Database: Source: Microsoft OLE DB Provider for SQL Server
Description: Login failed. The login is from an untrusted domain and cannot be used with Integrated authentication.
Number: -2147467259
Source: Microsoft OLE DB Provider for SQL Server
Nativeerror: 18452
SQLState: 42000
Bruce Zhang-MSFT 3,736 Reputation points

2022-01-19T01:49:58.86+00:00

Hi @Martin Larsen ,

As expected, the error message is indeed related to the SQL connection.

The next step to determine is whether IIS or SQL Server is not quickly applying the latest password. Can you manually recycle gMSA passwords? If possible, please log into the database through MS management immediately after recycling. Once you can't log in with the new password, we can be sure that it's a SQL problem.
Martin Larsen 1 Reputation point

2022-01-20T16:20:54.613+00:00

I havent been able find any command to recycle the gMSA password.
Is that possible?? I've found "Reset-ADServiceAccountPassword" but as far as I can see this only works with MSA account

Is it somehow possible to see when the password actually changed accross the servers??
Martin Larsen 1 Reputation point

2022-01-20T16:35:29.327+00:00

I found this in the Domain Controllers eventlog.
EventId 4776 at 09:08:03

The computer attempted to validate the credentials for an account.

Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Logon Account: sFeUdbyderInfo$
Source Workstation: FELINEWEB1
Error Code: 0xC000006A
Bruce Zhang-MSFT 3,736 Reputation points

2022-01-21T02:23:17.543+00:00

Hi @Martin Larsen ,

gMAS is not owned by Microsoft, so I don't know how to recycle it. Do you know when gMAS recycle password? If you know it, you can wait for next time.

IIS doesn't have feature to log app pool account password changed. But you can get this log from domain controller for event ID 627, 628, 4723 or 4724.

If your SQL and Server are on the same machine, event ID 4776 won't help much, it doesn't detail the exact password change time in SQL and IIS. Error Code: 0xC000006A means only Account logon with misspelled or bad password. So at the moment the easiest way seems to be that you log in to MS management immediately after password recovery. Being able to log in with the new password means that the problem is with IIS, and being able to log in with the old password means that the problem is with SQL Server.
Martin Larsen 1 Reputation point

2022-01-21T10:26:29.883+00:00
bit confused.
The Group Managed Service Accounts are a part of the Windows system.

I did a bit more digging.

A bit more information regarding our setup.

2 webservers (Web1 and Web2)

SQL Server

2 domain controllers

On the domain controllers i look in the "Application and Servives Logs / Directory Service"
and found eventids 2946 regarding this issue.

DC02 (Web1) - 17-1-2022 09:12:56 EventId 2946 A caller successfully fetched the password of a group managed service account. Group Managed Service Account Object: CN=sFeUdbyderInfo,[Removed]

DC01 (Web2) - 17-1-2022 09:02:00 EventId 2946 A caller successfully fetched the password of a group managed service account. Group Managed Service Account Object: CN=sFeUdbyderInfo,[Removed]

The errors occured on Web1 from 09:08:03 - 09:12:24 which matches on the interval above.
It seems like the Web1 didn't change the password until 9.12.56

Around 09.02.30 an eventid 2041 was also logged, (Duplicate event log entries were suppressed. )

It seems to me that the other servers detected the password has changed around 9.02 and then requested the new password, but Web1 did not make the same change until 9.12.56
Bruce Zhang-MSFT 3,736 Reputation points

2022-01-25T07:13:27.053+00:00

Hi @Martin Larsen ，

I want to know that how your server1 use gMAS account. Specifically, which part of IIS uses the gMAS account?

Second it how the IIS connect SQL Server. Sepcifically, what connection string is used by the application in IIS to connect to SQL, and what method is used to send the gMAS account or credentials to SQL for verification.

The main thing is how many applications in server1 need to update the gMAS account, and whether these applications update the account at the same time or sequentially.
Martin Larsen 1 Reputation point

2022-01-25T13:56:04.533+00:00

We use various connection-string configurations.
The ones that had failures this time were these 2:

Provider=SQLOLEDB.1;Packet Size=8192;Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=xxx;Data Source=yyy;

Data Source=xxx;Initial Catalog=yyy;Integrated Security=SSPI;Application Name=zzz;

The IIS is configured to use the gMSA account through the Application Pool identity settings. (Pipeline mode is integrated)

Right now 20 websites, each with its own account, are running on the webservers.
Therefore the update should only affect one account/application on the server.
Bruce Zhang-MSFT 3,736 Reputation points

2022-01-26T06:56:37.303+00:00

Hi @Martin Larsen ,

According to your description, IIS just use this account but never change it. This issue is more likely caused by the untimely update of the gMSA account in server1. GMSA is an entry in Windows Server Security Services. I have added the tag of Windows Server, once the engineer of Windows Server sees it, they will reply to you.