HTTPS access through TMG fails from a certain VLAN with a very unusual error: FWX_E_SEQ_ACK_MISMATCH

Article
06/15/2012

In this blog post, I’ll be talking about an interesting problem that I dealt with recently. The problem was that clients running in a certain VLAN were not able to establish HTTPS connections through TMG server. Due to the nature of the network, the clients should be configured as SecureNet clients (my customer cannot configure them as web proxy clients or use TMG client software because these machines are guest machines)

I asked for the usual data from our customer to find out what was happening during the problem:

- Network trace & HTTPWatch logs from a test client

- TMG data packager from the TMG server

1. After receiving the data, I started from client side. What I see on the client side is the client starts failing right after the initial TCP 3-way handshake:

Note: Please note that, for privacy purposes all IP addresses have been replaced with random addresses.

=> Client network trace:

(ip.addr eq 10.110.233.50 and ip.addr eq 172.16.1.1) and (tcp.port eq 50183 and tcp.port eq 443)

453 14:44:32 01.05.2012 27.1395045 0.0000000 10.110.233.50 172.16.1.1 TCP TCP: [Bad CheckSum]Flags=......S., SrcPort=50183, DstPort=HTTPS(443), PayloadLen=0, Seq=367515135, Ack=0, Win=8192 ( Negotiating scale factor 0x2 ) = 8192

456 14:44:32 01.05.2012 27.1565249 0.0170204 172.16.1.1 10.110.233.50 TCP TCP:Flags=...A..S., SrcPort=HTTPS(443), DstPort=50183, PayloadLen=2, Seq=2180033885 - 2180033888, Ack=367515136, Win=64240 ( Scale factor not supported ) = 64240

457 14:44:32 01.05.2012 27.1565443 0.0000194 10.110.233.50 172.16.1.1 TCP TCP: [Bad CheckSum]Flags=...A...., SrcPort=50183, DstPort=HTTPS(443), PayloadLen=0, Seq=367515136, Ack=2180033888, Win=64860 (scale factor 0x0) = 64860

458 14:44:32 01.05.2012 27.1585176 0.0019733 10.110.233.50 172.16.1.1 TLS TLS:TLS Rec Layer-1 HandShake: Client Hello.

465 14:44:32 01.05.2012 27.4744962 0.3159786 10.110.233.50 172.16.1.1 TCP TCP:[ReTransmit #458] [Bad CheckSum]Flags=...AP..., SrcPort=50183, DstPort=HTTPS(443), PayloadLen=127, Seq=367515136 - 367515263, Ack=2180033888, Win=64860 (scale factor 0x0) = 64860

476 14:44:33 01.05.2012 28.0828875 0.6083913 10.110.233.50 172.16.1.1 TCP TCP:[ReTransmit #458] [Bad CheckSum]Flags=...AP..., SrcPort=50183, DstPort=HTTPS(443), PayloadLen=127, Seq=367515136 - 367515263, Ack=2180033888, Win=64860 (scale factor 0x0) = 64860

494 14:44:34 01.05.2012 29.2948179 1.2119304 10.110.233.50 172.16.1.1 TCP TCP:[ReTransmit #458]Flags=...AP..., SrcPort=50183, DstPort=HTTPS(443), PayloadLen=127, Seq=367515136 - 367515263, Ack=2180033888, Win=64860 (scale factor 0x0) = 64860

512 14:44:35 01.05.2012 30.3815127 1.0866948 172.16.1.1 10.110.233.50 TLS TLS:Continued Data: 2 Bytes

513 14:44:35 01.05.2012 30.3815385 0.0000258 10.110.233.50 172.16.1.1 TCP TCP:Flags=...A...., SrcPort=50183, DstPort=HTTPS(443), PayloadLen=0, Seq=367515263, Ack=2180033888, Win=64860 (scale factor 0x0) = 64860

516 14:44:35 01.05.2012 30.5019345 0.1203960 10.110.233.50 172.16.1.1 TCP TCP:[ReTransmit #458]Flags=...AP..., SrcPort=50183, DstPort=HTTPS(443), PayloadLen=127, Seq=367515136 - 367515263, Ack=2180033888, Win=64860 (scale factor 0x0) = 64860

535 14:44:37 01.05.2012 31.7018989 1.1999644 10.110.233.50 172.16.1.1 TCP TCP:[ReTransmit #458] [Bad CheckSum]Flags=...AP..., SrcPort=50183, DstPort=HTTPS(443), PayloadLen=127, Seq=367515136 - 367515263, Ack=2180033888, Win=64860 (scale factor 0x0) = 64860

568 14:44:38 01.05.2012 33.1872833 1.4853844 172.16.1.1 10.110.233.50 TLS TLS:Continued Data: 6 Bytes

- Client cannot connect to remote web site because SSL/TLS negotiation doesn’t succeed because no response from the Web server is received (from client’s perspective)

2. Then I decided to check things from TMG server perspective. I first checked the network trace that was collected on the internal interface of TMG server through which the client request was received:

6457 14:33:49 01.05.2012 39.8915584 0.0000000 10.110.233.50 172.16.1.1 TCP TCP:Flags=......S., SrcPort=50183, DstPort=HTTPS(443), PayloadLen=0, Seq=367515135, Ack=0, Win=8192 ( Negotiating scale factor 0x2 ) = 8192

6461 14:33:49 01.05.2012 39.9079944 0.0164360 172.16.1.1 10.110.233.50 TCP TCP:Flags=...A..S., SrcPort=HTTPS(443), DstPort=50183, PayloadLen=0, Seq=2180033885, Ack=367515136, Win=64240 ( Scale factor not supported ) = 64240

6462 14:33:49 01.05.2012 39.9084181 0.0004237 10.110.233.50 172.16.1.1 TCP TCP:Flags=...A...., SrcPort=50183, DstPort=HTTPS(443), PayloadLen=0, Seq=367515136, Ack=2180033888, Win=64860 (scale factor 0x0) = 64860

6463 14:33:49 01.05.2012 39.9103936 0.0019755 10.110.233.50 172.16.1.1 TLS TLS:TLS Rec Layer-1 HandShake: Client Hello.

6489 14:33:49 01.05.2012 40.2259991 0.3156055 10.110.233.50 172.16.1.1 TCP TCP:[ReTransmit #6463]Flags=...AP..., SrcPort=50183, DstPort=HTTPS(443), PayloadLen=127, Seq=367515136 - 367515263, Ack=2180033888, Win=64860 (scale factor 0x0) = 64860

6614 14:33:50 01.05.2012 40.8343158 0.6083167 10.110.233.50 172.16.1.1 TCP TCP:[ReTransmit #6463]Flags=...AP..., SrcPort=50183, DstPort=HTTPS(443), PayloadLen=127, Seq=367515136 - 367515263, Ack=2180033888, Win=64860 (scale factor 0x0) = 64860

6806 14:33:51 01.05.2012 42.0457229 1.2114071 10.110.233.50 172.16.1.1 TCP TCP:[ReTransmit #6463]Flags=...AP..., SrcPort=50183, DstPort=HTTPS(443), PayloadLen=127, Seq=367515136 - 367515263, Ack=2180033888, Win=64860 (scale factor 0x0) = 64860

6872 14:33:52 01.05.2012 43.1311020 1.0853791 172.16.1.1 10.110.233.50 TCP TCP:Flags=...A..S., SrcPort=HTTPS(443), DstPort=50183, PayloadLen=0, Seq=2180033885, Ack=367515136, Win=64240 ( Scale factor not supported ) = 64240

6873 14:33:52 01.05.2012 43.1314010 0.0002990 10.110.233.50 172.16.1.1 TCP TCP:Flags=...A...., SrcPort=50183, DstPort=HTTPS(443), PayloadLen=0, Seq=367515263, Ack=2180033888, Win=64860 (scale factor 0x0) = 64860

6891 14:33:52 01.05.2012 43.2517820 0.1203810 10.110.233.50 172.16.1.1 TCP TCP:[ReTransmit #6463]Flags=...AP..., SrcPort=50183, DstPort=HTTPS(443), PayloadLen=127, Seq=367515136 - 367515263, Ack=2180033888, Win=64860 (scale factor 0x0) = 64860

7110 14:33:54 01.05.2012 44.4514449 1.1996629 10.110.233.50 172.16.1.1 TCP TCP:[ReTransmit #6463]Flags=...AP..., SrcPort=50183, DstPort=HTTPS(443), PayloadLen=127, Seq=367515136 - 367515263, Ack=2180033888, Win=64860 (scale factor 0x0) = 64860

7481 14:33:55 01.05.2012 45.9360680 1.4846231 172.16.1.1 10.110.233.50 TCP TCP:Flags=...A.R.., SrcPort=HTTPS(443), DstPort=50183, PayloadLen=0, Seq=2180033886, Ack=367515136, Win=8192 (scale factor 0x0) = 8192

- TMG server doesn’t really send responses back to the client

3. Then I decided to check the network trace collected on the external interface of the TMG server:

21 14:33:49 01.05.2012 39.6940232 0.0000000 10.110.235.202 172.16.1.1 TCP TCP:Flags=......S., SrcPort=55073, DstPort=HTTPS(443), PayloadLen=0, Seq=367515135, Ack=0, Win=8192 ( Negotiating scale factor 0x2 ) = 8192

22 14:33:49 01.05.2012 39.7097097 0.0156865 172.16.1.1 10.110.235.202 TCP TCP:Flags=...A..S., SrcPort=HTTPS(443), DstPort=55073, PayloadLen=0, Seq=2180033885, Ack=367515136, Win=64240 ( Scale factor not supported ) = 64240

23 14:33:52 01.05.2012 42.9329903 3.2232806 172.16.1.1 10.110.235.202 TCP TCP:Flags=...A..S., SrcPort=HTTPS(443), DstPort=55073, PayloadLen=0, Seq=2180033885, Ack=367515136, Win=64240 ( Scale factor not supported ) = 64240

24 14:33:55 01.05.2012 45.7379523 2.8049620 172.16.1.1 10.110.235.202 TCP TCP:Flags=...A.R.., SrcPort=HTTPS(443), DstPort=55073, PayloadLen=0, Seq=2180033886, Ack=367515136, Win=8192 (scale factor 0x0) = 8192

- External Web server sends a response to initial TCP 3-way handshake request that is forwarded by TMG, but TMG server doesn’t proceed with the connection

4. Then I checked the Web Proxy/Firewall log on the TMG server:

01.05.2012 14:33 Denied 10.110.233.50 50183 172.16.1.1 443 0xc0040034 FWX_E_SEQ_ACK_MISMATCH

When I check more details on that error code, I see that we fail because we receive a TCP packet with an invalid sequence number:

https://msdn.microsoft.com/en-us/library/ms812624.aspx/

FWX_E_SEQ_ACK_MISMATCH 0xC0040034 A TCP packet was rejected because it has an invalid sequence number or an invalid acknowledgement number.

So TMG Server drops the TCP ACK packet (3^rd TCP packet in TCP 3-way handshake) coming from the client because it has an invalid TCP ACK number

5. The problem was also visible in the ETL trace:

…

… handshake packet is dropped becuase ACK (2180033888) no equal ISN(peer)+1 (2180033886)

… Warning:The packet failed TCP sequence validation

… Warning:The packet is dropped because of SEQ_ACK_MISMATCH

…

6. When we check the TMG side network trace we see it there:

SequenceNumber: 367515135 (0x15E7D5FF)

SequenceNumber: 2180033885 (0x81F0AD5D)

AcknowledgementNumber: 367515136 (0x15E7D600)

SequenceNumber: 367515136 (0x15E7D600)

AcknowledgementNumber: 2180033888 (0x81F0AD60)

That Acknowledgement number SHOULD HAVE BEEN 2180033886 (0x81F0AD5E)

So TMG ignores the rest of the session (like TLS client hello coming from the client machine)

6463 14:33:49 01.05.2012 39.9103936 0.0019755 10.110.233.50 172.16.1.1 TLS TLS:TLS Rec Layer-1 HandShake: Client Hello.

7. When I check the client side trace, I see that the ACK number in the TCP ACK packet is really set to the wrong value (2180033888) by the client:

Frame: Number = 6462, Captured Frame Length = 60, MediaType = ETHERNET

+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[00-11-22-33-44-55],SourceAddress:[00-12-34-56-78-1B]

+ Ipv4: Src = 10.110.233.50, Dest = 172.16.1.1, Next Protocol = TCP, Packet ID = 24041, Total IP Length = 40

- Tcp: Flags=...A...., SrcPort=50183, DstPort=HTTPS(443), PayloadLen=0, Seq=367515136, Ack=2180033888, Win=64860 (scale factor 0x0) = 64860

SrcPort: 50183

DstPort: HTTPS(443)

SequenceNumber: 367515136 (0x15E7D600)

AcknowledgementNumber: 2180033888 (0x81F0AD60)

+ DataOffset: 80 (0x50)

+ Flags: ...A....

Window: 64860 (scale factor 0x0) = 64860

Checksum: 0x4B5D, Good

UrgentPointer: 0 (0x0)

8. One can think that it’s a problem with TCPIP stack on the client, but when we check the TCP SYN ACK packet (the second TCP packet sent by TMG server before the TCP ACK with wrong sequence number is sent by the client) we see that the client receives that TCP SYN ACK packet with 2 bytes extra data (which is something unusual for a TCP SYN ACK packet - such packets shouldn’t have data just should have TCP header):

Frame: Number = 456, Captured Frame Length = 60, MediaType = ETHERNET

+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[00-11-22-33-44-55],SourceAddress:[00-12-34-56-78-1B]

+ DestinationAddress: Microsoft Corporation [00-11-22-33-44-55]

+ SourceAddress: Test Data [00-12-34-56-78-1B]

EthernetType: Internet IP (IPv4), 2048(0x800)

- Ipv4: Src = 172.16.1.1, Dest = 10.110.233.50, Next Protocol = TCP, Packet ID = 1342, Total IP Length = 46

+ Versions: IPv4, Internet Protocol; Header Length = 20

+ DifferentiatedServicesField: DSCP: 0, ECN: 0

TotalLength: 46 (0x2E)

Identification: 1342 (0x53E)

+ FragmentFlags: 0 (0x0)

TimeToLive: 120 (0x78)

NextProtocol: TCP, 6(0x6)

Checksum: 46957 (0xB76D)

SourceAddress: 194.53.208.72

DestinationAddress: 10.110.233.50

- Tcp: Flags=...A..S., SrcPort=HTTPS(443), DstPort=50183, PayloadLen=2, Seq=2180033885 - 2180033888, Ack=367515136, Win=64240 ( Scale factor not supported ) = 64240

SrcPort: HTTPS(443)

DstPort: 50183

SequenceNumber: 2180033885 (0x81F0AD5D)

AcknowledgementNumber: 367515136 (0x15E7D600)

+ DataOffset: 96 (0x60)

+ Flags: ...A..S.

Window: 64240 ( Scale factor not supported ) = 64240

Checksum: 0x365C, Good

UrgentPointer: 0 (0x0)

- TCPOptions:

- MaxSegmentSize: 1

type: Maximum Segment Size. 2(0x2)

OptionLength: 4 (0x4)

MaxSegmentSize: 1380 (0x564)

- TCPPayload: SourcePort = 443, DestinationPort = 50183

UnknownData: Binary Large Object (2 Bytes)

That’s why the TCPIP stack running on the client sends a TCP ACK number which is 2 more than the value that should be:

ACK number sent by the client: AcknowledgementNumber: 2180033888 (0x81F0AD60)

The correct ACK number that should have been sent: AcknowledgementNumber: 2180033886 (0x81F0AD5E)

9. And when we check the TCP SYN ACK packet that is leaving the TMG server, we don’t see such an extra 2 bytes:

Frame: Number = 6461, Captured Frame Length = 60, MediaType = ETHERNET

+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[00-11-22-33-44-55],SourceAddress:[00-12-34-56-78-1B]

+ Ipv4: Src = 172.16.1.1, Dest = 10.110.233.50, Next Protocol = TCP, Packet ID = 1342, Total IP Length = 44

- Tcp: Flags=...A..S., SrcPort=HTTPS(443), DstPort=50183, PayloadLen=0, Seq=2180033885, Ack=367515136, Win=64240 ( Scale factor not supported ) = 64240

SrcPort: HTTPS(443)

DstPort: 50183

SequenceNumber: 2180033885 (0x81F0AD5D)

AcknowledgementNumber: 367515136 (0x15E7D600)

+ DataOffset: 96 (0x60)

+ Flags: ...A..S.

Window: 64240 ( Scale factor not supported ) = 64240

Checksum: 0x365E, Good

UrgentPointer: 0 (0x0)

- TCPOptions:

- MaxSegmentSize: 1

type: Maximum Segment Size. 2(0x2)

OptionLength: 4 (0x4)

MaxSegmentSize: 1380 (0x564)

So that 2 bytes extra data is somehow added to TCP SYN ACK by something else (like the NIC driver on the TMG server, a network device running in between (like Wireless access point etc) or the NIC driver on the client machine

SUMMARY:

==========

In summary the HTTPS connectivity problem stems from an issue between the Client and TMG server (including the NIC layer or below on the client and server and the network devices/links in between the two)

My customer informed me that the issue was visible with any clients which makes it unlikely that it’s a client side issue. I advised my customer to update NIC drivers on the TMG server side and check the network devices running in the path and upgrade firmwares where possible.

Hope this helps

Thanks,

Murat

HTTPS access through TMG fails from a certain VLAN with a very unusual error: FWX_E_SEQ_ACK_MISMATCH

Additional resources