Event Metadata Overview

This is an overview of what to expect for the end-to-end journey of an event for each of the four tracing technologies furnished by Microsoft: TraceLogging, Manifest-based, WPP, and MOF. It approaches the problem with respect to both an individual event's payload and the accompanying metadata that describes its structure, making it possible to later decode the event into sensible pieces of data. Understanding the journey of each piece of an event is important when choosing which tracing technology is appropriate for a project. There is no one-size-fits-all solution to tracing.

Event Payloads

For any of the Microsoft-provided tracing technologies, when calling the Write command (for instance EventWrite for manifest-based or TraceLoggingWrite for TraceLogging), the data provided for the event at runtime is folded into a contiguous binary blob called the event payload. This is separate from the event schema or event metadata (discussed below), which describes the layout of this binary blob for decoding tools. How exactly the generation of the event payload happens varies depending on the tracing technology used and ultimately whether the event is classic (TraceEvent style) or modern (EventWrite style).

For classic events, the flag in EVENT_TRACE_HEADER is passed into TraceEvent which determines how this occurs:

- If WNODE_FLAG_USE_MOF_PTR is specified, an array of MOF_FIELDS follows the EVENT_TRACE_HEADER in memory. The ETW runtime will concatenate the event data as specified in these fields to form the event payload.

- If WNODE_FLAG_USE_MOF_PTR is not specified, the event payload is constructed by the user directly in memory following the EVENT_TRACE_HEADER.

Both MOF and WPP are classic providers. However, the WPP implementation takes care of all of this for you.

For non-classic events, a number of EVENT_DATA_DESCRIPTORS are passed into EventWrite. The ETW runtime will concatenate the event data as specified in these descriptors to form the event payload.

Both Manifest-based and TraceLogging technologies are generally modern providers. With manifest-based, if you let mc.exe generate logging code for you (um or km switch), payload generation is taken care of for you. Payload generation is always taken care of by TraceLogging.

The end result is that a payload for an event is generated at runtime. All payloads are fundamentally similar in that they are binary blobs of data containing only the field data for that particular event, regardless of the tracing technology used and which field types are supported by that tracing technology. Without the event metadata, the event payload is meaningless and unintelligable.

The ETW runtime's duty is then to carry this event blob from the provider to the consumer, where it is re-associated with its metadata and becomes decodable. The ETW runtime will add a bit of extra metadata indicating the circumstances of the payload -- for instance things like the timestamp, thread ID, and keywords of the event. This information is relevant to how the event was routed through the runtime, and it is also useful for understanding the identity and context of the event at decoding time. It is eventually surfaced as part of the EVENT_TRACE or EVENT_RECORD for the consumer

Event Metadata

The journey for event metadata is different depending on which tracing technology is used, and it is one of the largest considerations when comparing each technology.

MOF

Event metadata is authored by hand by the creator of the event in the form of a special MOF schema in a .mof file. The schema authored must match the payload that is logged in order for the event to be correctly decoded by consumers. Event decoders require that the .mof file is registered on the system using mofcomp.exe. For more information on publishing MOF schemas, see Publishing Your Event Schema for a Classic Provider.

WPP

Event metadata is automatically generated and stored in your binary's .pdb by tracewpp.exe during the build process. This metadata can later be extracted in the form of a standalone .tmf file by way of tracepdb.exe. Event decoders typically require either the .pdb or the .tmf file. For more information on tracepdb.exe, see Tracepdb Overview, and for more information on tracewpp.exe, see TraceWPP task.

Manifest-based

Event definitions are authored in a .man file. Event manifests are run through the manifest compiler (mc.exe), generating the headers needed for source compilation as well as several .bin binary resource files. These resource files must then be compiled as resources into a binary and the resulting binary placed on the system that intends to decode events from that manifest-based provider. Additionally, the manifest must be installed on the system using wevtutil.exe Most importantly, the resourceFileName and messageFileName attributes on the provider XML element in the installed event manifest must correctly identify the full absolute path to the binary in which the resource files were placed. Note that these paths may be overwritten at manifest install time by using the wevtutil.exe switches /rf and /mf. A system that has not correctly and fully performed these steps will be unable to decode events from this provider. Event decoders thus require an installed manifest, and the binary with the associated resources installed in the location specified by resource and message file paths. For more information on publishing manifest-based schemas, see Publishing Your Event Schema for a Manifest-based Provider.

TraceLogging

All TraceLogging events are self-describing. The event metadata necessary to decode the event is automatically generated and transmitted along with the payload in a compact binary format. Event decoders need only the TraceLogging event and an understanding of the TraceLogging metadata format.

Configuring TDH for Decoding

There are many decoding tools out there. However, it is recommended that you use TDH where possible, as it decodes all tracing technologies with a unified API. Depending on the type of tracing you are interested in decoding, TDH may require explicit configuration.

MOF

TDH utilizes MOF decoding data registered on the system using mofcomp.exe. For more information, see Publishing Your Event Schema for a Classic Provider.

WPP

TDH must be pointed towards either the .pdb file or the associated .tmf for the WPP provider you are trying to decode. This can be performed by calling TdhSetDecodingParameter. Otherwise, the decoding engine will fall back on the environment variable TRACE_FORMAT_SEARCH_PATH.

Manifest-based

TDH utilizes all manifest-based providers registered on the system using wevtutil.exe.

TraceLogging

The newest versions of TDH automatically detect and decode TraceLogging events with no additional steps.