The client and server follow the same series of steps to encrypt a block of data. First, a MAC value is generated over the unencrypted data.
Pad1 = 0x36 repeated 40 times to give 320 bits Pad2 = 0x5C repeated 48 times to give 384 bits SHAComponent = SHA(MACKeyN + Pad1 + DataLength + Data) MACSignature = First64Bits(MD5(MACKeyN + Pad2 + SHAComponent)) MACKeyN is either MACKey40, MACKey56 or MACKey128, depending on the negotiated key strength.
DataLength is the size of the data to encrypt in bytes, expressed as a little-endian 32-bit integer. Data is the information to be encrypted. The first 8 bytes of the generated MD5 hash are used as an 8-byte MAC value to send on the wire.
Next, the data block is encrypted with RC4 using the current client or server encryption substitution table. The encrypted data is appended to the 8-byte MAC value in the network packet.
Decryption involves a reverse ordering of the previous steps. First, the data is decrypted using the current RC4 decryption substitution table. Then, a 16-byte MAC value is generated over the decrypted data, and the first 8 bytes of this MAC are compared to the 8-byte MAC value that was sent over the wire. If the MAC values do not match, an appropriate error is generated and the connection is dropped.