A Sandwich You Should Avoid Ordering
Secure by default is a great response to the security problems that plagued Microsoft nearly a decade ago, but sometimes we, as consumers of a product, confuse the concepts of vulnerability and functionality. The receive connector is an example that I recently revisited, causing me enough confusion with a customer that I felt it might be time to collect and then share my thoughts on what exactly these things are. As described by TechNet, "Receive connectors are configured on computers that are running Microsoft Exchange Server 2010 and that have the Hub Transport server role or Edge Transport server role installed. Receive connectors represent a logical gateway through which all inbound messages are received." So, how else might we define these in a succinct manner? I would propose, "A receive connector is a logical gateway that assumes responsibility for message delivery or relay, based on whether a sender matches the connector's configured rules."
By default, Exchange 2010 ships with a Default [ServerName] receive connector on every machine with the hub transport role. One of the things that makes Exchange 2010 secure by default is that the default receive connector will not accept anonymous requests, will require at least one form of authentication, and will prefer TLS when possible. While this ensures internal message delivery across the SMTP protocol within your organization, it simultaneously prevents any messages from entering your environment from the Internet (arguably defeating the purpose of email). Email can enter your organization any number of ways, but one of the more common scenarios is through a hosted message security & hygiene service, like Microsoft's Forefront Online Protection for Exchange (FOPE). In this scenario, all email is referred to FOPE though the MX records. It is delivered to and scanned by the Microsoft service and relayed directly to your organization (either through an Edge role, or TMG/UAG-published hub transport role). Most customers (in its simplest, but rarest configuration) simply stand up a second receive connector designed to listen only to the FOPE address lists and allow anonymous authentication. That's what we had in this story, well mostly.
The customer had a TMG Firewall Sandwich that looked something like this:
SMTP-->FOPE-->Firewall Public-->NAT-->TMG Published SMTP-->Firewall (same one) Internal-->Load Balancer SMTP-->Hub Transport Role Server
What made it tricky was the security team's insistence that the internet-facing NIC on the TMG server be assigned a private address which was then published via Network Address Translation (NAT) behind the "real firewall." I won't even begin to discuss why this is a ridiculous design, but it was what we had to work with. So a SMTP Server publishing rule was created on the TMG server and directed to the Virtual IP (VIP) of the load balancer. The rule was designed to make SMTP traffic appear to come from the original sender (versus a proxy by the TMG server). Initial tests showed that the connection from FOPE was successfully reaching the TMG Server, and being passed through to one of the Hub Transport Servers. However, our TMG logs showed that no SYN/ACK was being received by the FOPE server and the connection was being closed. The log file on the TMG Server looked similar to the following:
A connection was closed because no SYN/ACK reply was received from the server.
A little research confirmed my suspicions that blamed the firewall sandwich. Because outbound traffic inside the network was not configured to route back through the TMG, but instead directly through the "real firewall," all hopes of communication were lost on different highways. The quick and dirty fix, of course, is to change the configuration of the TMG rule to specify that Requests appear to come from the TMG Server. This solved the first problem, but mail still wasn't flowing. This made it necessary to crack open NetMon. Below is a synopsis of what we saw:
SMTP:Rsp 250 -(REMOVED)CAS01.customer.com Hello [IP REMOVED], 260 bytes
SMTP:Cmd STARTTLS, Server is currently able to negotiate the use of TLS
SMTP:Rsp 220 2.0.0 SMTP server ready, 29 bytes
SSL:SSLv2RecordLayer, Error (needs reassembly)
SSL:SSLv2RecordLayer, Client Finished (needs reassembly)
SSL:SSLv2RecordLayer, ClientHello (0x01) (needs reassembly)
SSL:SSLv2RecordLayer, Error (needs reassembly)
Basically, the connection was getting established, but any attempts to establish an SSL connection (TLS) were failing and the connection was being closed with a reset (RST) flag on the TMG logs. This is easily blamed on NAT. If TMG is going to negotiate security with FOPE, they need to be able to verify sending addresses, with which NAT interferes. And this is where receive connectors come into play. Our original internet receive connector was expecting everything to come from the TMG server, since it was terminating and replaying connections as part of its proxy duties. So, it was set up to allow port 25 from the IP Addresses assigned to the TMG server with anonymous access and default authentication mechanisms. Once the initial failures started, the thought was to disable TLS (all auth methods, actually) and that should solve our immediate problem. But it didn't.
I could see on the TMG that traffic was flowing to the VIP of the load balancer and SMTP traffic was answering from either of the two HT servers. So I installed NetMon on the HT server and watched both machines at the same time. What I discovered was that the TMG server was sending to the VIP of the load balancer, which was configured for SMTP load distribution. The load balancer was establishing a new session with the hub transport server. So the hub replied to the load balancer which replied to the TMG server. So my receive connector was never even getting used because it was expecting traffic from the TMG server that never arrived. What was happening is that all my traffic was going to the default receive connector that was still trying to use TLS and no anonymous. By now, I was wondering, how do I even know which receive connector I am talking to? There's an att(ribute) for that. Using the following cmdlet, you can establish a specialized banner message on your receive connector that will show up on the SMTP:220 response in either a telnet session or on the NetMon trace. The cmdlet is:
get-ReceiveConnector "[ServerName]\[ConnectorName]" | Set-ReceiveConnector -Banner "220 CAS01Default"
Your banner can say anything, but it should identify the server and connector in some way. Once establishing these for my primary receive connectors in question, I turned back on the NetMon trace and validated that, sure enough, all my traffic was by-passing the TMG receive connector in favor of the default, where it just failed. My solution was to adjust the address range of the TMG connector so that it contained the VIP of the hardware load balancer. Immediately, the SMTP traffic started flowing, end-to-end on all machines being monitored.
One word of caution. This is not an optimal solution, just because mail is again flowing. FOPE is TLS capable, as is this customer's environment. However, in the name of security, the security team "broke" the standard communication protocols at layers 3 and 4 of the OSI model. This ultimately prevented secure transmissions at layer 7 (TLS in Exchange). So, while mail is once again flowing, it is actually doing it in a less secure manner than if the firewall sandwich had just been abandoned as a security principle.