Event 1035, MSExchangeTransport - Inbound authentication failed with error UnexpectedExchangeAuthBlob for Receive connector

  • Article

Scenario

Ran into an interesting case today where mail flow had quit working in one direction between sites. The environment was on Exchange 2010. Basically, Site A and Site B could not send mail to an exchange server in Site C. They could send to each other, and mail was flowing out from Site C, but inbound mail to Site C was queueing up.

The error in the Application event logs looked as follows:

Log Name:      Application
Source:        MSExchangeTransport
Date:          8/24/2016 1:21:43 PM
Event ID:      1035
Task Category: SmtpReceive
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:     SiteCserver.contoso.com
Description:
Inbound authentication failed with error UnexpectedExchangeAuthBlob for Receive connector Default SiteCServer. The authentication mechanism is ExchangeAuth. The source IP address of the client who tried to authenticate to Microsoft Exchange is [IP of Site A exchange server (or site B)].
Event Xml:
<Event xmlns="https://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MSExchangeTransport" />
    <EventID Qualifiers="32772">1035</EventID>
    <Level>3</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2016-08-24T17:21:43.000000000Z" />
    <EventRecordID>10489441</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SiteCserver.contoso.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>UnexpectedExchangeAuthBlob</Data>
    <Data>Default SiteCServer</Data>
    <Data>ExchangeAuth</Data>
    <Data>[IP of Site A exchange server (or Site B)]</Data>
  </EventData>
</Event>

There was no time skew in the environment and no backpressure events were showing up (Event IDs 15001 -15007) in the Application event logs.

Resolution

Normally, I would spend a great deal of time working through Protocol logs and network traces to figure out what is happening here, but luckily for me, I have an engineer named Miguel Ortiz on my team and he has seen this before.

In this particular case, what was happening was, the Kerberos token from Site A and Site B destined for Site C had gotten corrupted. Site C had suffered a severe outage and we believe it was during this time that the Kerberos tokens became corrupted. We went through the following steps to remediate the issue:

    1. On the Site C domain controllers, we restarted the Kerberos Key Distribution Center service (KDC).
    2. On the Site C Exchange server we purged the Kerberos tickets using the KList Purge command from an administrative command prompt
      1. klist purge
    3. Next, we restarted the Netlogon service on the Site C Exchange server.
      1. Net Stop Netlogon && Net Start Netlogon
    4. Then we restarted the Transport Service on the Site C Exchange server.
      1. Net Stop MSExchangeTransport && Net Start MSExchangeTransport
    5. After transport came back up on Site C, we then repeated steps 2 through 4 on the Site A and Site B servers.

This process basically clears out the Kerberos tickets and forces Site A and Site B to re-send their Kerberos request to Site C. In other words, they cleared the corrupted token and requested a new one.

Once this process completed, mail flow returned to normal and no further 1035 events were thrown.

Special Thanks

Thanks to Miguel Ortiz for helping me get through this case quickly and working out the steps to resolve it.