Unable to Access a Published Apache Server behind ISA Server using SSL

Consider a scenario where you are publishing a third party web server, in this case an Apache Server that uses HTTPS through ISA Server 2006. Randomly the site doesn’t work, clients are unable to access it and when this happens the publishing rule test button shows the result below (error 0x80090326):

SSLEncryptAlert_ErroronISA

Notice that this error talks about a server certificate error, so clearly it is something during the SSL process.

Reviewing the Data

Using ISA Data Packager in repro mode (web proxy / web publishing template) was possible to collect simultaneous netmon traces from both NICs (internal and external). During those traces it was possible to see that the SSL handshake on the external interface using the certificate that was bound to the Web Listener on ISA was working just fine. Reviewing the SSL Handshake on the internal interface, while ISA was negotiating with the published server (Apache) we had a failure. Here it is the moment of the failure:

ISA APACHE TCP TCP:Flags=......S., SrcPort=24433, DstPort=443, PayloadLen=0, Seq=3108278462, Ack=0, Win=65535 ( ) = 65535

APACHE ISA TCP TCP:Flags=...A..S., SrcPort=443, DstPort=24433, PayloadLen=0, Seq=2120534540, Ack=3108278463, Win=5840 ( Scale factor not supported ) = 5840

ISA APACHE TCP TCP: Flags=...A...., SrcPort=24433, DstPort=443, PayloadLen=0, Seq=3108278463, Ack=2120534541, Win=65535 (scale factor 0x0) = 65535

After finishing the TCP Handshake they start the SSL handshake and this is done by ISA sending the SSL Client Hello as shown below:

ISA APACHE TLS TLS:TLS Rec Layer-1 HandShake

TLSSSLData: Transport Layer Security (TLS) Payload Data

- TLS: TLS Rec Layer-1 HandShake

- TlsRecordLayer: TLS Rec Layer-1 HandShake

ContentType: HandShake

- Version: TLS 1.0

Major: 3 (0x3)

Minor: 1 (0x1)

Length: 88 (0x58)

- SSLHandshake: SSL HandShake ClientHello(0x01)

HandShakeType: ClientHello(0x01)

Length: 84 (0x54)

- ClientHello: TLS 1.0

+ Version: TLS 1.0

+ RandomBytes:

SessionIDLength: 16 (0x10)

SessionID: Binary Large Object (16 Bytes)

CipherSuitesLength: 22

+ TLSCipherSuites: TLS_RSA_WITH_RC4_128_MD5 { 0x00,0x04 }

+ TLSCipherSuites: TLS_RSA_WITH_RC4_128_SHA { 0x00,0x05 }

+ TLSCipherSuites: TLS_RSA_WITH_3DES_EDE_CBC_SHA { 0x00,0x0A }

+ TLSCipherSuites: TLS_RSA_WITH_DES_CBC_SHA { 0x00,0x09 }

+ TLSCipherSuites: TLS_NTRU_NSS_WITH_AES_256_CBC_SHA { 0x00, 0x64 }

+ TLSCipherSuites: TLS_NTRU_NSS_WITH_3DES_EDE_CBC_SHA { 0x00, 0x62 }

+ TLSCipherSuites: TLS_RSA_EXPORT_WITH_RC4_40_MD5 { 0x00,0x03 }

+ TLSCipherSuites: TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5 { 0x00,0x06 }

+ TLSCipherSuites: TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA { 0x00,0x13 }

+ TLSCipherSuites: TLS_DHE_DSS_WITH_DES_CBC_SHA { 0x00,0x12 }

+ TLSCipherSuites: TLS_NTRU_NSS_WITH_AES_128_CBC_SHA { 0x00, 0x63 }

CompressionMethodsLength: 1 (0x1)

CompressionMethods: 0 (0x0)

ExtensionsLength: 5 (0x5)

Right after that Apache sends a SSL Encrypt Alert error as shown below:

APACHE ISA TLS TLS:TLS Rec Layer-1 Encrypted Alert

TLSSSLData: Transport Layer Security (TLS) Payload Data

- TLS: TLS Rec Layer-1 Encrypted Alert

- TlsRecordLayer: TLS Rec Layer-1 Encrypted Alert

ContentType: Encrypted Alert

- Version: TLS 1.0

Major: 3 (0x3)

Minor: 1 (0x1)

Length: 2 (0x2)

EncryptedData: Binary Large Object (2 Bytes)

15 03 01 00 02 02 2F

....../

By looking to the last two bytes of the hex value under Encrypted data we can find the meaning of the alert:

o 2F in Hex = 47 in Decimal

o 47 maps to illegal_parameter(47) error according to TLS RFC (https://www.ietf.org/rfc/rfc2246.txt?number=2246)

Note: thanks to sudeepg for this nice approach reading SSL Encrypt Alert.

Apache Server is saying that the TLS SSL Client Hello sent by ISA as an illegal parameter for this SSL negotiation.

Conclusion

This problem can happen because MS10-049 which is installed on ISA Server. As a temp workaround this update was removed and the issue got resolved on this particular scenario. However, ultimately you should not remove this update; you should talk to the third party company web server admin and discuss the CVE-2009-3555 with him and how their product adequate for that. If you are publishing an IIS Server you might have this issue too, if you do, read the post below to see how to fix it:

https://blogs.msdn.com/b/jpsanders/archive/2010/09/08/understanding-problems-with-ms10-049-kb-980436-and-ietf-rfc5746.aspx