question

CoryC-7854 avatar image
0 Votes"
CoryC-7854 asked CoryCandia-4545 edited

TLS Auth causes account login failure audits (Event ID 36879)

Exchange Experts,

I can't eliminate an 'account failed to log on' audit caused by exchange's TLS auth mechanism. Everytime I get an email delivered to the server via our receive connector, the server tries to match the sender's cert using NTLM (I think).

I can't fix it regardless of the security options I select on the receive connector. It looks like exchange's TLS is trying to map the sending server's certificate to a machine or system account, which won't work. It must be a built in mechanism of Windows TLS?

Can anyone help get this to stop trying to match it to system accounts or point me in the right direction?

EDIT: Environment: Server 2019, Exchange 2019 CU8

References


  1. AUDIT LOG ENTRY ON EXCHANGE SERVER



    An account failed to log on.

    Subject:
    Security ID: S-1-0-0
    Account Name: -
    Account Domain: -
    Logon ID: 0x0

    Logon Type: 3

    Account For Which Logon Failed:
    Security ID: S-1-0-0
    Account Name:
    Account Domain:

    Failure Information:
    Failure Reason: An Error occured during Logon.
    Status: 0xC000006D
    Sub Status: 0x80090325

    Process Information:
    Caller Process ID: 0x0
    Caller Process Name: -

    Network Information:
    Workstation Name: -
    Source Network Address: -
    Source Port: -

    Detailed Authentication Information:
    Logon Process: Schannel
    Authentication Package: Microsoft Unified Security Protocol Provider
    Transited Services: -
    Package Name (NTLM only): -
    Key Length: 0

    This event is generated when a logon request fails. It is generated on the computer where access was attempted.

    The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

    The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).

    The Process Information fields indicate which account and process on the system requested the logon.

    The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

    The authentication information fields provide detailed information about this specific logon request.
    - Transited services indicate which intermediate services have participated in this logon request.
    - Package name indicates which sub-protocol was used among the NTLM protocols.
    - Key length indicates the length of the generated session key. This will be 0 if no session key was requested.



  2. SCHANNEL LOG IN SYSTEM LOG

    The certificate received from the remote client application was not successfully mapped to a client system account. The error code is 0xC000006D. This is not necessarily a fatal error, as the server application may still find the certificate acceptable.

System Log / Source schannel / Event ID 36879



  1. EXCHANGE RECEIVE CONNECTOR SETTINGS



    RunspaceId : 84e2514a-91f6-4fee-ab6f-3acfe6e4edb8
    AuthMechanism : Tls
    Banner :
    BinaryMimeEnabled : True
    Bindings : {[::]:25, 0.0.0.0:25}
    ChunkingEnabled : True
    DefaultDomain :
    DeliveryStatusNotificationEnabled : True
    EightBitMimeEnabled : True
    SmtpUtf8Enabled : True
    BareLinefeedRejectionEnabled : False
    DomainSecureEnabled : True
    EnhancedStatusCodesEnabled : True
    LongAddressesEnabled : False
    OrarEnabled : False
    SuppressXAnonymousTls : False
    ProxyEnabled : False
    AdvertiseClientSettings : False
    Fqdn : mail.c3.%MyFQDM%.com
    ServiceDiscoveryFqdn :
    TlsCertificateName : <I>CN=Sectigo RSA Domain Validation Secure Server CA, O=Sectigo Limited, L=Salford, S=Greater Manchester,
    C=GB<S>CN=mail.c3.%MyFQDM%.com
    Comment :
    Enabled : True
    ConnectionTimeout : 00:10:00
    ConnectionInactivityTimeout : 00:05:00
    MessageRateLimit : Unlimited
    MessageRateSource : IPAddress
    MaxInboundConnection : 5000
    MaxInboundConnectionPerSource : 20
    MaxInboundConnectionPercentagePerSource : 2
    MaxHeaderSize : 256 KB (262,144 bytes)
    MaxHopCount : 60
    MaxLocalHopCount : 12
    MaxLogonFailures : 3
    MaxMessageSize : 36 MB (37,748,736 bytes)
    MaxProtocolErrors : 5
    MaxRecipientsPerMessage : 200
    PermissionGroups : AnonymousUsers
    PipeliningEnabled : True
    ProtocolLoggingLevel : Verbose
    RemoteIPRanges : {65.55.88.0/24, 94.245.120.64/26, 207.46.51.64/26, 207.46.163.0/24, 213.199.154.0/24, 213.199.180.128/26,
    2a01:111:f400:7c00::/54, 23.103.132.0/22, 23.103.136.0/21, 104.47.0.0/17, 23.103.198.0/23, 23.103.200.0/21,
    2a01:111:f400:fc00::/54, 65.55.169.0/24, 134.170.140.0/24, 134.170.171.0/24...}
    RequireEHLODomain : False
    RequireTLS : True
    EnableAuthGSSAPI : False
    ExtendedProtectionPolicy : None
    LiveCredentialEnabled : False
    TlsDomainCapabilities : {mail.protection.outlook.com:AcceptCloudServicesMail}
    Server : C3-CARM-EXCH
    TransportRole : FrontendTransport
    RejectReservedTopLevelRecipientDomains : False
    RejectReservedSecondLevelRecipientDomains : False
    RejectSingleLabelRecipientDomains : False
    AcceptConsumerMail : False
    SizeEnabled : Enabled
    TarpitInterval : 00:00:05
    AuthTarpitInterval : 00:00:05
    MaxAcknowledgementDelay : 00:00:30
    AdminDisplayName :
    ExchangeVersion : 0.1 (8.0.535.0)
    Name : Default Frontend C3-CARM-EXCH
    DistinguishedName : CN=Default Frontend C3-CARM-EXCH,CN=SMTP Receive Connectors,CN=Protocols,CN=C3-CARM-EXCH,CN=Servers,CN=Exchange
    Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=C3,CN=Microsoft
    Exchange,CN=Services,CN=Configuration,DC=C3,DC=%MyFQDM%,DC=LOCAL
    Identity : C3-CARM-EXCH\Default Frontend C3-CARM-EXCH
    Guid : ccd41531-2e07-4158-b033-c8e0a2e82aa7
    ObjectCategory : C3.%MyFQDM%.LOCAL/Configuration/Schema/ms-Exch-Smtp-Receive-Connector
    ObjectClass : {top, msExchSmtpReceiveConnector}
    WhenChanged : 3/19/2021 11:42:50 AM
    WhenCreated : 2/28/2021 5:07:17 PM
    WhenChangedUTC : 3/19/2021 3:42:50 PM
    WhenCreatedUTC : 2/28/2021 10:07:17 PM
    OrganizationId :
    Id : C3-CARM-EXCH\Default Frontend C3-CARM-EXCH
    OriginatingServer : C3-CARM-DC.C3.%MyFQDM%.LOCAL
    IsValid : True
    ObjectState : Unchanged



  2. SMTP PROTOCOL LOG

    220 mail.c3.%MyFQDN%.com Microsoft ESMTP MAIL Service ready at Fri, 19 Mar 2021 11:34:52 -0400
    EHLO NAM04-CO1-obe.outbound.protection.outlook.com
    250 mail.c3.%MyFQDN%.com Hello [104.47.45.54] SIZE 37748736 PIPELINING DSN ENHANCEDSTATUSCODES STARTTLS 8BITMIME BINARYMIME CHUNKING SMTPUTF8
    STARTTLS
    220 2.0.0 SMTP server ready
    CN=mail.c3.%MyFQDN%.com CN=Sectigo RSA Domain Validation Secure Server CA, O=Sectigo Limited, L=Salford, S=Greater Manchester, C=GB 0083FCA8E4520287D092B34DFB87FC2C6C 6185A6B241DE02FCD5DEAA247130D947F912D04F 2020-04-14T20:00:00.000Z 2022-07-17T20:00:00.000Z mail.c3.%MyFQDN%.com;autodiscover.%MyFQDN%.com;www.%MyFQDN%.com Sending certificate Subject Issuer name Serial number Thumbprint Not before Not after Subject alternate names
    CN=mail.protection.outlook.com, O=Microsoft Corporation, L=Redmond, S=Washington, C=US CN=DigiCert Cloud Services CA-1, O=DigiCert Inc, C=US 0700204CC3051A9ED4F65599C48E742C 5D440E75FE29C4D5A223314BB0EC1DCBEB1A0903 2021-02-05T19:00:00.000Z 2022-02-05T18:59:59.000Z mail.protection.outlook.com;.mail.eo.outlook.com;.mail.protection.outlook.com;mail.messaging.microsoft.com;outlook.com;.olc.protection.outlook.com;.pamx1.hotmail.com Remote certificate Subject Issuer name Serial number Thumbprint Not before Not after Subject alternate names
    TLS protocol SP_PROT_TLS1_2_SERVER negotiation succeeded using bulk encryption algorithm CALG_AES_256 with strength 256 bits, MAC hash algorithm CALG_SHA_384 with strength 0 bits and key exchange algorithm CALG_ECDH_EPHEM with strength 384 bits
    EHLO NAM04-CO1-obe.outbound.protection.outlook.com
    CN=mail.protection.outlook.com, O=Microsoft Corporation, L=Redmond, S=Washington, C=US CN=DigiCert Cloud Services CA-1, O=DigiCert Inc, C=US 0700204CC3051A9ED4F65599C48E742C 5D440E75FE29C4D5A223314BB0EC1DCBEB1A0903 2021-02-05T19:00:00.000Z 2022-02-05T18:59:59.000Z mail.protection.outlook.com;.mail.eo.outlook.com;.mail.protection.outlook.com;mail.messaging.microsoft.com;outlook.com;.olc.protection.outlook.com;.pamx1.hotmail.com Validated received certificate (cached) Subject Issuer name Serial number Thumbprint Not before Not after Subject alternate names
    TlsDomainCapabilities='AcceptCloudServicesMail'; Status='Success'; Domain='mail.protection.outlook.com'
    SMTPSubmit SMTPAcceptAnySender SMTPAcceptAuthoritativeDomainSender AcceptRoutingHeaders Set Session Permissions
    250 mail.c3.%MyFQDN%.com Hello [104.47.45.54] SIZE 37748736 PIPELINING DSN ENHANCEDSTATUSCODES 8BITMIME BINARYMIME CHUNKING SMTPUTF8 XOORG
    MAIL FROM:<ccandia@%MyFQDN%.com> SIZE=33012 XOORG=%MyFQDN%.com
    SMTPAcceptAnyRecipient SMTPAcceptAuthoritativeDomainSender BypassAntiSpam BypassMessageSizeLimit Granted Mail Item Permissions
    08D8EAE82C60FCAA;2021-03-19T15:34:53.220Z;1 receiving message
    RCPT TO:<tkickass@%MyFQDN%.com>
    250 2.1.0 Sender OK
    250 2.1.5 Recipient OK
    BDAT 22895 LAST
    Set mail item OORG to '%MyFQDN%.com' based on 'MAIL FROM:'
    Proxy destination(s) obtained from OnProxyInboundMessage event. Correlation Id:ddffa6c1-5c87-43e1-aea4-2e6f2dff102b
    250 2.6.0 <DM6PR10MB3163472928DF2ABFA297AF61B9689@DM6PR10MB3163.namprd10.prod.outlook.com> [InternalId=1679332212747, Hostname=C3-CARM-EXCH.C3.%MyFQDN%.LOCAL] 24433 bytes in 0.147, 161.380 KB/sec Queued mail for delivery
    QUIT
    221 2.0.0 Service closing transmission channel



  3. What Microsoft says about it:
    dn786445(v=ws.11)







office-exchange-server-administrationoffice-exchange-server-mailflow
eventlog.png (35.2 KiB)
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

So, Im confused. :)

Why does the inbound Office 365 connector have all that checked?

All you need is TLS and anonymous enabled for 365 Hybrid.
The TLS enforcement is handled by the receive connector :
TlsDomainCapabilities : {mail.protection.outlook.com:AcceptCloudServicesMail}

80207-image.png


0 Votes 0 ·
image.png (21.0 KiB)

Any luck figuring this out? I'm battling the same issues with these security event logs causing issues with our alerting/monitoring.

0 Votes 0 ·

The solution was a a compromise:

We eliminated my custom connector and reverted back to the default connector. It completely eliminated the error message. It's incredibly frustrating that I couldn't figure out how to separate out the O365 traffic, but since we firewall SMTP and don't allow any external or partner direct SMTP connections except Office 365 (which is filtered), it works for my use case.

A lot of people didn't understand the real issue, which wasn't that exchange worked fine and we could ignore the message or disable schannel events (which this is not, so the error message remains). The real issue in my case was ignoring the message means ignoring logs that the IPS system is going to lose its mind about since Windows thinks it's an invalid login. In the current world, who is going to get away with configuring systems not to report hundreds/thousands of logs of invalid log in attempts? Can you imagine explaining that after the breach/data loss? You wouldn't, you'd be too busy looking for a new job ;)

0 Votes 0 ·
ZhengqiLou-MSFT avatar image
0 Votes"
ZhengqiLou-MSFT answered

Hi @CoryC-7854 ,

Sorry that I don't quite understand the issue, and you said this caused the accounts logon failure, logging on to OWA/ECP or clients?
And the mail flow is good?
If you don't mind, restore the Connectors to the default status and check. And I'm thinking the Exchange Server is necessary member of the group list.
80138-image.png

Regards,
Lou


If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.


image.png (123.2 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

CoryC-7854 avatar image
0 Votes"
CoryC-7854 answered AndyDavid converted comment to answer

Mail flow is fine, partially. Mail flows in and out of the environment.

The issue is specific to SMTP delivery using TLS.

When our upstream sending server (office 365) connects to the on prem exchange server, we require TLS. When SMTP does the TLS process and the certificates are exchanged, it works and allows encrypted mail transfer, but Windows Server 2019 seems to try and use the sending server's certificate attempting to match a local or domain user account. That's what causes the login failure audit.

We don't want exchange to attempt to match another SMTP server's certificate to an account, we just want the SMTP session encrypted. I didn't notice this behavior on Server 2012 R2 with Exchange 2013. Not saying it wasn't doing it, but we never got the monitoring system blowing up about failed logins, so I am guessing not.

P.S. When you do TLS, you need to match the public certificate name like mail.FQDN.com, which will not work when you select 'Exchange Server Authentication'. I reset the receive connector back to mostly original. We have tried all different options trying to prevent the user account cert match (event ID 36879)
80199-receiveconnectorsecurity.png



5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

CoryC-7854 avatar image
0 Votes"
CoryC-7854 answered ZhengqiLou-MSFT commented

Dude! I know, right! (In agreement with you Andy). I shouldn't have to have any of that other stuff checked, but even Microsoft scripted tech support wants all that crap on there. I doesn't make any sense to me either.

More importantly, I have also tried exactly that configuration, but it still gives the same errors. Very frustrating.

This is a quote from the Microsoft site about the error (event 36879):

When a server application requires client authentication, Schannel automatically attempts to map the certificate supplied by the client to a user account.

I think Windows Server / Exchange is passing the sending server's cert to LSASS trying to match an account. I can't prove that, but it's what the blurb says, and it makes sense with the error messages. I just can't find anything about it, or turn it off. I can't imagine there is something that unique about my installation of Server 2019 / Exchange CU8. That has to happen to other people right?

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @CoryC-7854 ,

Oh that was my fault, so sorry for the misunderstanding, I was think it's a full on-prem environment and didn't realize that is for the hybrid.

And well, I want to know if this warning event has any exact effects for your server, is it causing the "logon error"? or only the audit logs because I found a similar warning that could be ignored, that is a stupid question but it helps me understand what the point is.

Also if you want to disable the Certificate Mapping Authentication, you could Client Certificate Mapping Authentication and CertificateMappingMethods
Hope that could help.

Regards,
Lou

0 Votes 0 ·

That was a very clever suggestion which I didn't think about. Sadly, it didn't change anything. I think the Transport service or Front End Transport Service is doing the certificate mapping attempt, not the web/iss/W3SVC. I can't figure out how to prove this though. The login failure event process ID maps back so lsass.exe (which makes sense), I just can't see what process ID tries to initiate the client certificate map.

SMTP is listening on 25 which maps back to the front end transport service/exe.

0 Votes 0 ·

Hi @CoryC-7854 ,

Have you ever checked the certificate? Since the O365 could work correctly, I would think this is because of the certificate you're using on your on-prem Exchange server.

Also I'm still trying to find anything useful, but I'd like to ask that is there any actual error like logon OWA failed or just the failure event logs?

Regards,
Lou

0 Votes 0 ·
AndyDavid avatar image
0 Votes"
AndyDavid answered CoryC-7854 commented

You are going to hate this suggestion.
Add this registry key and silence the events :)

Set-itemproperty HKLM:\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel -Name EventLogging -Value 0

I mean, otherwise, its working right?

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HA! 'I like where your head's at', or 'great minds', etc...

I had that same thought on day two. The SCHANNEL event log is not really the problem. The main problem is along with the SCHANNEL event log is the login failure event, which is the actual problem.

The login failure log blows up all the monitoring systems with false positives since there's one for every time an SMTP server initiates TLS, so the more mail delivery, the more login failures. You can imagine how it looks for an IDS/Monitor...

Bottom line, turning off schannel logs didn't solve the overall problem. Thanks for the continued suggestions.

0 Votes 0 ·