New in V1.2: Default Window Output Policies

  • Article

Who knows what window output policies in StreamInsight LINQ are? Anyone? Well, you have been using them whenever you have used a window. Output policies specify the timestamps of the result of the aggregation that happens on the window. Even if you didn’t really care about that, you had to specify the output policy parameter for the window. In V1.2 we are introducing

  • Default policies so that you don’t have to specify them for every window, and
  • Another output policy for Hopping/Tumbling windows.

Importantly, there is no breaking change. Whatever you used in 1.1 is still working in 1.2, with the same behavior. For count- and snapshot-windows, the single existing output policy is becoming the default:

SnapshotWindow() is equivalent to
SnapshotWindow(SnapshotWindowOutputPolicy.Clip)

CountByStartTimeWindow(num) is equivalent to
CountByStartTimeWindow(num, CountWindowOutputPolicy.PointAlignToWindowEnd)

Set-based operations on top of snapshot windows always produce an event with the timestamps of the snapshot window itself, while operations on count windows produce a point event at the end of the window.

The only existing output policy for hopping and tumbling windows in StreamInsight 1.1 was to produce the window size as result event lifetime. Now this turned out to be impractical in many cases, especially with overlapping hopping windows. Have a look at the following diagram:

image

Just like the windows, also the result events overlap. When using this result in a subsequent operation, like a join with another stream, this easily leads to unexpected results. In many cases, only a single aggregation result at each point in time is needed (instead of four concurrently valid payloads in the diagram above). To this end, we have introduced another output policy for hopping (and tumbling) windows, which produces a point event at the end of the window:

image

This policy is now also the default for hopping windows:

HoppingWindow(size, hop) is equivalent to
HoppingWindow(size, hop, HoppingWindowOutputPolicy.PointAlignToWindowEnd)

TumblingWindow(size) is equivalent to
TumblingWindow(size, HoppingWindowOutputPolicy.PointAlignToWindowEnd)

This can now easily be converted into a continuous signal, for instance by using the ToSignal() macro that we discussed earlier. This signal has a payload at each point in time and hence is suitable to be joined with another stream, which is not necessarily synchronized in time. Typical example: join the result of a hopping aggregation with some reference stream:

var result = from a in avg.ToSignal(e => e.SourceId)
             from r in reference.ToSignal(e => e.SourceId)
             select new { alarm = a.Value < r.Threshold };

Dealing with point events instead of overlapping intervals in the context of windows helps avoiding unexpected side-effects and—most of the times—corresponds to the intended semantics.

Regards,
The StreamInsight Team