MS-OXCFXICS - How to parse the FastTransfer Stream

  • Article

Note: This article was written using version 16.2 (10/30/2014) of the MS-OXCFXICS document as reference and all links contained in this article reference sections of that version of the document. The current version of the MS-OXCFXICS document can be found here: https://msdn.microsoft.com/en-us/library/cc463916(v=exchg.80).aspx

Resources:

Introduction:
The FastTransfer Stream as described in MS-OXCFXICS section 2.2.4 and its related subsections can be an intimidating adversary if you are trying to implement a mail client or server that uses RPC as its primary method of communication (as opposed to ActiveSync or Exchange Web Services). The important thing to understand is that it is nothing more than a serialized collection of markers, properties, and values.

Sample Data:
The following sample data is the first part of a much larger stream of bytes. However, for the purposes of this article, it contains enough different property types to give you a good idea of how it all works.

0x0000: | 03 00 12 40 02 01 E1 65 00 00 00 00 02 01 E0 65
0x0010: | 16 00 00 00 52 F6 85 EC 7D 43 2E 4A A9 60 34 50
0x0020: | 88 53 D9 0A 00 00 00 00 03 F5 40 00 08 30 00 D0
0x0030: | 9E A8 81 85 D0 01 02 01 E2 65 16 00 00 00 52 F6
0x0040: | 85 EC 7D 43 2E 4A A9 60 34 50 88 53 D9 0A 00 00
0x0050: | 00 00 20 7C 02 01 E3 65 17 00 00 00 16 52 F6 85
0x0060: | EC 7D 43 2E 4A A9 60 34 50 88 53 D9 0A 00 00 00
0x0070: | 00 20 7C 1F 00 01 30 0C 00 00 00 49 00 4E 00 42
0x0080: | 00 4F 00 58 00 00 00 14 00 49 67 01 00 00 00 00
0x0090: | 00 03 F4 03 00 39 66 FB 03 00 00 40 00 07 30 80
0x00A0: | 39 06 A8 81 85 D0 01 0B 00 F4 10 00 00

Breaking it Down:
The Lexical Structure of the FastTransfer Stream might look complicated at first but we can summarize it as follows:

  1. The stream consists of elements. 
  2. Each element is either a marker or a propValue. 
    1. If the element matches an entry from the table in section 2.2.4.1.4, it’s a Marker. If not, it's a propValue.
  3. propValues are either fixed or variable length.
    1. The first 2 bytes of the propValue represent the data type. Reference MS-OXCDATA section 2.11.1 to find out what type it is and whether it's fixed or variable length.
      1. If it's fixed length, read the next x number of bytes as indicated by the type.
      2. If it's variable length, follow the description provided to correctly read the value.

Markers:
Markers are the easiest to understand and come in 2 types. They are either stand-alone, or consist of a pair of start and end markers. They are nothing more than a 4-byte value that tells you something about the properties that come after it in the stream (unless it's an end marker). Taking a look at our example data stream from above let's look at the first element and try to determine if it's a marker or a propValue.

Element[0]:
0x0000: | 03 00 12 40 02 01 E1 65 00 00 00 00 02 01 E0 65

The value of the first element is 0x40120003 (remember that int32 type values are transmitted in little-endian format). From looking at the table in MS-OXCDATA section 2.2.4.1.4 we can see that this represents the IncrSyncChange Marker which "signifies the start of ICS information pertaining to the message." That's all there is to it. Moving on…

propValues:
If the element isn't a Marker (the element value is NOT in the table in MS-OXCFXICS section 2.2.4.1.4), it's a propValue. There are 2 types of propValues, fixed and variable length. The table in MS-OXCDATA section 2.11.1 contains all of the possible types. The list is fairly long so I will only talk about the ones that appear in our sample data. Let's take a look at the first one…

Element[1]:
0x0000: | 03 00 12 40 02 01 E1 65 00 00 00 00 02 01 E0 65

The element value is 0x65E10102, which is not a Marker (check the table!), which means it's a propValue. There are 2 things that you need to do with this. First, look up the type in MS-OXCDATA section 2.11.1. In this case, the type 0x0102 means that the property is a PtypBinary. According to the description, this type is "Variable size; a COUNT field followed by that many bytes." The COUNT field is 4 bytes in length and should be read as an integer, which in this case is the value 0x00000000, or simply 0. This also means that this is the end of this propType. Because the COUNT value is 0, no bytes follow it that belong to this property. But what does the property represent? That's the second thing to do, look up the Property ID in MS-OXPROPS. In this case, the Property ID 0x65E1 belongs to the PidTagParentSourceKey property (section 2.850).

Taking a look at the next element…

Element[2]:
0x0000: | 03 00 12 40 02 01 E1 65 00 00 00 00 02 01 E0 65
0x0010: | 16 00 00 00 52 F6 85 EC 7D 43 2E 4A A9 60 34 50
0x0020: | 88 53 D9 0A 00 00 00 00 03 F5 40 00 08 30 00 D0

You can probably tell by how many bytes I've highlighted above that there is definitely more going on with this next one. Following the same process as before, get the element (0x65E00201) which is a propValue. This is also a PtypBinary type so you know that the next thing you have to do is get the COUNT, which in this case is 0x00000016, or 22 in decimal format. This tells us that the next 22 bytes should be read as a byte array. That's easy, but what does that data actually represent? Going into that much depth with each property would be beyond the scope of this article, but here's a hint: Look up the Property ID (0x65E0) in MS-OXPROPS (PidTagSourceKey) and then check the defining reference (MS-OXCFXICS section 2.2.1.2.5).

Let's parse a few more propValues…

Element[3]:
0x0020: | 88 53 D9 0A 00 00 00 00 03 F5 40 00 08 30 00 D0
0x0030: | 9E A8 81 85 D0 01 02 01 E2 65 16 00 00 00 52 F6

propValue: 0x30080040
Property type: 0x0040 - PtypTime
Property ID: 0x3008 - PidTagLastModificationTime
Value: 05/03/2015 09:15:12
Note: See MS-DTYP section 2.3.3 on how to interpret this type of value.

Element[4]:
0x0030: | 9E A8 81 85 D0 01 02 01 E2 65 16 00 00 00 52 F6
0x0040: | 85 EC 7D 43 2E 4A A9 60 34 50 88 53 D9 0A 00 00
0x0050: | 00 00 20 7C 02 01 E3 65 17 00 00 00 16 52 F6 85

propValue: 0x65E20102
Property type: 0x0102 - PtypBinary
Property ID: 0x65E2 - PidTagChangeKey
Value: See MS-OXCFXICS section 2.2.1.2.7

Element[5]:
0x0050: | 00 00 20 7C 02 01 E3 65 17 00 00 00 16 52 F6 85
0x0060: | EC 7D 43 2E 4A A9 60 34 50 88 53 D9 0A 00 00 00
0x0070: | 00 20 7C 1F 00 01 30 0C 00 00 00 49 00 4E 00 42

propValue: 0x65E30102
Property type: 0x0102 - PtypBinary
Property ID: 0x65E3 - PidTagPredecessorChangeList
Value: See MS-OXCFXICS section 2.2.1.2.8

Element[6]:
0x0070: | 00 20 7C 1F 00 01 30 0C 00 00 00 49 00 4E 00 42
0x0080: | 00 4F 00 58 00 00 00 14 00 49 67 01 00 00 00 00

propValue: 0x3001001F
Property type: 0x001F - PtypString
Property ID: 0x3001 - PidTagDisplayName
Value: "INBOX"
Note: Per MS-OXCDATA section 2.11.1 this "a string of Unicode characters in UTF-16LE format encoding with terminating null character (0x0000)."

Element[7]:
0x0080: | 00 4F 00 58 00 00 00 14 00 49 67 01 00 00 00 00
0x0090: | 00 03 F4 03 00 39 66 FB 03 00 00 40 00 07 30 80

propValue: 0x67490014
Property type: 0x0014 - PtypInteger64
Property ID: 0x6749 - PidTagParentFolderId
Value: -863846703525003263

Element[8]:
0x0090: | 00 03 F4 03 00 39 66 FB 03 00 00 40 00 07 30 80

propValue: 0x66390003
Property type: 0x0003 - PtypInteger32
Property ID: 0x6639 - PidTagRights
Value: 0x000003FB
Note: See MS-OXCPERM section 2.2.7 on how to interpret these flags.

Element[9]:
0x0090: | 00 03 F4 03 00 39 66 FB 03 00 00 40 00 07 30 80
0x00A0: | 39 06 A8 81 85 D0 01 0B 00 F4 10 00 00

propValue: 0x30070040
Property type: 0x0040 - PtypTime
Property ID: 0x3007 - PidTagCreationTime
Value: 05/03/2015 09:15:11
Note: See MS-DTYP section 2.3.3 on how to interpret this type of value.

Element[10]:
0x00A0: | 39 06 A8 81 85 D0 01 0B 00 F4 10 00 00

propValue: 0x10F4000B
Property type: 0x000B - PtypBoolean
Property ID: 0x3007 - PidTagAttributeHidden
Value: 0x0000 (False)
Note: If you look at the table in MS-OXCDATA section 2.11.1 you will see that the description for PtypBoolean states that it is "1 byte; restricted to 1 or 0." However, you can see that I have 2 bytes highlighted. This is because PtypBoolean is serialized differently than is described there. This difference is described in MS-OXCFXICS in section 2.2.4.1.3 where it states that the PtypBoolean type is "2-bytes in FastTransfer streams, instead of 1-byte as specified in [MS-OXCDATA]. Using little-endian byte ordering, "01 00" for TRUE and "00 00" for FALSE."

I hope this helps you to understand how the Fast Transfer stream is serialized. Even though we only looked at a few of the elements in this stream, it should give you a good idea on how to proceed to parse a complete FastTransfer stream.