question

JoeHinkle-7763 avatar image
0 Votes"
JoeHinkle-7763 asked JoeHinkle-7763 answered

Exchange 2016 Old Equipment Can't Email After CU21 Upgrade

We were lucky to have a good AV in place when we got hit with the most recent 0 day exploit for the web shell. We caught it immediately and in the end decided to completely rebuild our server to CU21 w/ the latest SU. We ran into some issues with IIS failing to install correctly and had to pay MS to help us fix it.

Now we have an odd issue where some 3rd party devices can't email when authenticating. Doesn't matter if we use TLS, SSL, or none, they won't work. Since we run an old AS400 and RPG doesn't support current standards I decided to piggy back on my AS400 Anon Receive connector and allow these devices to email by using IP restrictions and our noreply account as the reply address. This worked for most devices.

The problem is that I have one last device I can't get working. It's a really old Generator Adapter that only runs on old installs of IE/silverlight (we have this setup for just this purpose). I can't get the adapter to send emails either via auth or anonymously like our other other devices. I also can't find any error messages in the Generator Adapter, Exchange Logs, Event Viewer Logs, or Message Tracking Logs. What I really need is to just find the error message to see if I can fix this. Does anyone know of a good way to find the error between client and server? Or is there a way to tell what rule a IP is hitting when it tries to email? I'm ok with fixing auth or using anon rules by IP but neither seem to work on this old thing.

office-exchange-server-administrationoffice-exchange-server-mailflowoffice-exchange-server-connectivity
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

AndyDavid avatar image
1 Vote"
AndyDavid answered JoeHinkle-7763 commented

Did you enable protocol logging on the receive connectors?
Search for the IP of the sending device and see what the conversation looks like. If if doesnt exist, then you know its never even getting to the Exch Server
https://docs.microsoft.com/en-us/exchange/mail-flow/connectors/configure-protocol-logging?view=exchserver-2019#use-the-eac-to-configure-protocol-logging

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you for the response. Some connectors had it enabled, some didn't. I've enabled it on all the relevant ones and have done some testing with/without auth and I'm not seeing it hit the logs yet. I'm going to keep testing and see if I can get it to show up.

0 Votes 0 ·

Is there a telnet client on the Generator Adapter by chance?
If so, you can telnet on port 25 to the Exchange Server and see if you can send a message that way

0 Votes 0 ·

Not that I've found. This adapter is a Cummins PC550. The setup screen only has a few options for server, port, reply to, email, user, pass, and a drop down box with tls/ssl/none options. The test button just says "Test email has been sent" when you click it. The logs don't tell me anything and it acts like everything is working.

0 Votes 0 ·
JoeHinkle-7763 avatar image
0 Votes"
JoeHinkle-7763 answered KaelYao-MSFT commented

I've done some more digging and was able to find more information in the logs by using the windows Greg utility. I found the below information in the send and receive logs. This would be the adapter running in the morning when the generator tests. The email has a line of output that includes timestamps for start/stop time. If I'm reading this correctly it authed and connected. But I have no record of any emails or errors after that.

I changed out domain info to hide company information.

2021-09-15T11:45:49.005Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,0,10.73.0.46:25,10.73.0.152:51194,+,,
2021-09-15T11:45:49.005Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,1,10.73.0.46:25,10.73.0.152:51194,>,"220 exchange.my.server Microsoft ESMTP MAIL Service ready at Wed, 15 Sep 2021 07:45:48 -0400",
2021-09-15T11:45:49.529Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,2,10.73.0.46:25,10.73.0.152:51194,<,HELO Gener-01,
2021-09-15T11:45:49.529Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,3,10.73.0.46:25,10.73.0.152:51194,>,250 exchange.my.server Hello [10.73.0.152],
2021-09-15T11:45:49.555Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,4,10.73.0.46:25,10.73.0.152:51194,<,STARTTLS,
2021-09-15T11:45:49.555Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,5,10.73.0.46:25,10.73.0.152:51194,>,220 2.0.0 SMTP server ready,
2021-09-15T11:45:49.555Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,6,10.73.0.46:25,10.73.0.152:51194,, CN=exchange CN=exchange 2F7A001EDE7361984751FC6EBBD00559 B228395F1F65E5F36305159D7CA8CCFDC4D8D20F 2021-08-24T20:30:23.000Z 2026-08-24T20:30:23.000Z exchange;exchange.my.server,Sending certificate Subject Issuer name Serial number Thumbprint Not before Not after Subject alternate names
2021-09-15T11:45:49.671Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,7,10.73.0.46:25,10.73.0.152:51194,
,,"TLS protocol SP_PROT_TLS1_0_SERVER negotiation succeeded using bulk encryption algorithm CALG_3DES with strength 168 bits, MAC hash algorithm CALG_SHA1 with strength 160 bits and key exchange algorithm CALG_RSA_KEYX with strength 2048 bits"
2021-09-15T11:45:49.877Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,8,10.73.0.46:25,10.73.0.152:51194,<,EHLO Gener-01,
2021-09-15T11:45:49.877Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,9,10.73.0.46:25,10.73.0.152:51194,,,Client certificate chain validation status: 'EmptyCertificate'
2021-09-15T11:45:49.877Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,10,10.73.0.46:25,10.73.0.152:51194,
,,TlsDomainCapabilities='None'; Status='NoRemoteCertificate'
2021-09-15T11:45:49.877Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,11,10.73.0.46:25,10.73.0.152:51194,>,250 exchange.my.server Hello [10.73.0.152] SIZE 37748736 PIPELINING DSN ENHANCEDSTATUSCODES AUTH NTLM LOGIN X-EXPS GSSAPI NTLM 8BITMIME BINARYMIME CHUNKING XRDST,
2021-09-15T11:45:49.959Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,12,10.73.0.46:25,10.73.0.152:51194,<,AUTH LOGIN,
2021-09-15T11:45:49.959Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,13,10.73.0.46:25,10.73.0.152:51194,>,334 <authentication response>,
2021-09-15T11:45:49.988Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,14,10.73.0.46:25,10.73.0.152:51194,>,334 <authentication response>,
2021-09-15T11:45:50.065Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,15,10.73.0.46:25,10.73.0.152:51194,,SMTPSubmit SMTPAcceptAnyRecipient BypassAntiSpam AcceptRoutingHeaders,Set Session Permissions
2021-09-15T11:45:50.065Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,16,10.73.0.46:25,10.73.0.152:51194,
,mydomain/noreply,authenticated
2021-09-15T11:45:50.065Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,17,10.73.0.46:25,10.73.0.152:51194,,,ASyncBackendLocator.BeginGetDatabaseToServerMappingInfo for user No.Reply@mydomain.com.
2021-09-15T11:45:50.079Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,18,10.73.0.46:25,10.73.0.152:51194,
,,AsyncBackendLocator.EndGetDatabaseToServerMappingInfo for user No.Reply@mydomain.com
2021-09-15T11:45:50.079Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,19,10.73.0.46:25,10.73.0.152:51194,,,Setting up client proxy session to destination(s): exchange.my.server
2021-09-15T11:45:50.102Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,20,10.73.0.46:25,10.73.0.152:51194,
,,Proxy session was successfully set up. Session for mydomain/noreply will now be proxied
2021-09-15T11:45:50.102Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,21,10.73.0.46:25,10.73.0.152:51194,>,235 2.7.0 Authentication successful,
2021-09-15T11:45:50.662Z,EXCHANGE\Default Frontend EXCHANGE,08D96953AE78CD21,22,10.73.0.46:25,10.73.0.152:51194,-,,Local

2021-09-15T11:45:50.079Z,Client Proxy Send Connector,08D96953AE78CD22,0,,10.73.0.46:465,,,attempting to connect. Client proxy session for mydomain/noreply. Proxied session id 08D96953AE78CD21
2021-09-15T11:45:50.079Z,Client Proxy Send Connector,08D96953AE78CD22,1,10.73.0.46:23202,10.73.0.46:465,+,,
2021-09-15T11:45:50.080Z,Client Proxy Send Connector,08D96953AE78CD22,2,10.73.0.46:23202,10.73.0.46:465,<,"220 exchange.my.server Microsoft ESMTP MAIL Service ready at Wed, 15 Sep 2021 07:45:49 -0400",
2021-09-15T11:45:50.080Z,Client Proxy Send Connector,08D96953AE78CD22,3,10.73.0.46:23202,10.73.0.46:465,>,EHLO exchange.my.server,
2021-09-15T11:45:50.080Z,Client Proxy Send Connector,08D96953AE78CD22,4,10.73.0.46:23202,10.73.0.46:465,<,250 exchange.my.server Hello [10.73.0.46] SIZE 37748736 PIPELINING DSN ENHANCEDSTATUSCODES STARTTLS X-ANONYMOUSTLS AUTH GSSAPI NTLM X-EXPS GSSAPI NTLM 8BITMIME BINARYMIME CHUNKING XEXCH50 XRDST XSHADOWREQUEST,
2021-09-15T11:45:50.080Z,Client Proxy Send Connector,08D96953AE78CD22,5,10.73.0.46:23202,10.73.0.46:465,>,X-ANONYMOUSTLS,
2021-09-15T11:45:50.080Z,Client Proxy Send Connector,08D96953AE78CD22,6,10.73.0.46:23202,10.73.0.46:465,<,220 2.0.0 SMTP server ready,
2021-09-15T11:45:50.085Z,Client Proxy Send Connector,08D96953AE78CD22,7,10.73.0.46:23202,10.73.0.46:465,
," CN=.mydomain.com, OU=Domain Control Validated CN=Go Daddy Secure Certificate Authority - G2, OU=http://certs.godaddy.com/repository/, O=""GoDaddy.com, Inc."", L=Scottsdale, S=Arizona, C=US 4D1F8B7003B5958C 0739A684F526B3AEC5EA9675F12D06D09B85F055 2020-05-26T10:14:34.000Z 2022-03-08T16:03:40.000Z .mydomain.com;mydomain.com",Remote certificate Subject Issuer name Serial number Thumbprint Not before Not after Subject alternate names
2021-09-15T11:45:50.085Z,Client Proxy Send Connector,08D96953AE78CD22,8,10.73.0.46:23202,10.73.0.46:465,,,"TLS protocol SP_PROT_TLS1_2_CLIENT negotiation succeeded using bulk encryption algorithm CALG_AES_128 with strength 128 bits, MAC hash algorithm CALG_SHA_256 with strength 0 bits and key exchange algorithm CALG_ECDH_EPHEM with strength 256 bits"
2021-09-15T11:45:50.085Z,Client Proxy Send Connector,08D96953AE78CD22,9,10.73.0.46:23202,10.73.0.46:465,
,0739A684F526B3AEC5EA9675F12D06D09B85F055,Received certificate Thumbprint
2021-09-15T11:45:50.085Z,Client Proxy Send Connector,08D96953AE78CD22,10,10.73.0.46:23202,10.73.0.46:465,," CN=.mydomain.com, OU=Domain Control Validated CN=Go Daddy Secure Certificate Authority - G2, OU=http://certs.godaddy.com/repository/, O=""GoDaddy.com, Inc."", L=Scottsdale, S=Arizona, C=US 4D1F8B7003B5958C 0739A684F526B3AEC5EA9675F12D06D09B85F055 2020-05-26T10:14:34.000Z 2022-03-08T16:03:40.000Z .mydomain.com;mydomain.com",Proxy target certificate Subject Issuer name Serial number Thumbprint Not before Not after Subject alternate names
2021-09-15T11:45:50.085Z,Client Proxy Send Connector,08D96953AE78CD22,11,10.73.0.46:23202,10.73.0.46:465,>,EHLO exchange.my.server,
2021-09-15T11:45:50.086Z,Client Proxy Send Connector,08D96953AE78CD22,12,10.73.0.46:23202,10.73.0.46:465,<,250 exchange.my.server Hello [10.73.0.46] SIZE 37748736 PIPELINING DSN ENHANCEDSTATUSCODES AUTH GSSAPI NTLM LOGIN X-EXPS EXCHANGEAUTH GSSAPI NTLM X-EXCHANGEAUTH SHA256 8BITMIME BINARYMIME CHUNKING XEXCH50 XRDST XSHADOWREQUEST XPROXY XPROXYFROM X-MESSAGECONTEXT ADRC-2.1.0.0 EPROP-1.2.0.0 XSYSPROBE XORIGFROM XMESSAGEVALUE,
2021-09-15T11:45:50.087Z,Client Proxy Send Connector,08D96953AE78CD22,13,10.73.0.46:23202,10.73.0.46:465,>,X-EXPS EXCHANGEAUTH SHA256 ,
2021-09-15T11:45:50.087Z,Client Proxy Send Connector,08D96953AE78CD22,14,10.73.0.46:23202,10.73.0.46:465,>,<Binary Data>,
2021-09-15T11:45:50.088Z,Client Proxy Send Connector,08D96953AE78CD22,15,10.73.0.46:23202,10.73.0.46:465,<,235 <authentication information>,
2021-09-15T11:45:50.089Z,Client Proxy Send Connector,08D96953AE78CD22,16,10.73.0.46:23202,10.73.0.46:465,>,XPROXY SID=08D96953AE78CD21 IP=10.73.0.152 PORT=51194 DOMAIN=Gener-01 CAPABILITIES=0 SECID=Uy0xLTUtMjEtNjgzNDI5NjYtMTQ2NzQ1Mjg5My05Mjg0OTk0MzMtNjAxMA+3D+3D,
2021-09-15T11:45:50.102Z,Client Proxy Send Connector,08D96953AE78CD22,17,10.73.0.46:23202,10.73.0.46:465,<,250 XProxy accepted and authenticated,
2021-09-15T11:45:50.102Z,Client Proxy Send Connector,08D96953AE78CD22,18,10.73.0.46:23202,10.73.0.46:465,
,,Proxy session successfully set up for mydomain/noreply. Inbound session will now be blindly proxied
2021-09-15T11:45:50.662Z,Client Proxy Send Connector,08D96953AE78CD22,19,10.73.0.46:23202,10.73.0.46:465,-,,Remote

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I dont see any actual SMTP conversation with MAIL FROM or RCPT TO, etc....

0 Votes 0 ·

That's where I'm stuck. I can't figure out what's happening if I can't find an error, or the rest of the process to send an email.

I'm trying to work with our facilities guys to get the electrician to work with us. I think the best case scenario is that I can flash the firmware on this thing to get a more current release. I'm not getting any answers from anyone yet but one of the screens I found has a timestamp for the system of 2006. If that's the actual hardware age then I think we may be due for an upgrade.

0 Votes 0 ·

Hi,

From the SMTP log, it seems this device didn't send SMTP requests to the Exchange server.
If it is the case, I am afraid that it may not be possible to troubleshoot this issue on server side.

0 Votes 0 ·
JoeHinkle-7763 avatar image
0 Votes"
JoeHinkle-7763 answered

I ended up getting a firmware upgrade from the vendor and although it updated the UI to work in a modern browser it did not really update any of the features. The configs are identical and options are all the same. I still am unable to get it to work.

I was however able to get it to work with gmail using TLS. Since this is a one off problem for this specific old piece of equipment I'm considering this resolved and moving on. I still never figured out what it is about this server with the latest CU21 patch that is causing issues with 3rd party equipment but my workaround work.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.