parse operator

Evaluates a string expression and parses its value into one or more calculated columns. The calculated columns will have nulls, for unsuccessfully parsed strings. For more information, see the parse-where operator.

T | parse Text with "ActivityName=" name ", ActivityType=" type

Syntax

T | parse [kind=regex [flags=regex_flags] |simple|relaxed] Expression with * (StringConstant ColumnName [: ColumnType]) *...

Arguments

  • T: The input table.

  • kind:

    • simple (the default): StringConstant is a regular string value and the match is strict. All string delimiters should appear in the parsed string, and all extended columns must match the required types.

    • regex: StringConstant may be a regular expression and the match is strict. All string delimiters, which can be a regex for this mode, should appear in the parsed string, and all extended columns must match the required types.

    • flags: Flags to be used in regex mode like U (Ungreedy), m (multi-line mode), s (match new line \n), i (case-insensitive) in RE2 flags.

    • relaxed: StringConstant is a regular string value and the match is relaxed. All string delimiters should appear in the parsed string, but extended columns may partially match the required types. Extended columns that didn't match the required types will get the value null.

  • Expression: An expression that evaluates to a string.

  • ColumnName: The name of a column to assign a value to, extracted from the string expression.

  • ColumnType: Optional. The scalar value that indicates the type to convert the value to. The default is the string type.

Returns

The input table, extended according to the list of columns that are provided to the operator.

Tips

  • Use project if you also want to drop or rename some columns.

  • Use * in the pattern, to skip junk values.

    Note

    The * can't be used after a string type column.

  • The parse pattern may start with ColumnName and not only with StringConstant.

  • If the parsed Expression isn't of type string, it will be converted to type string.

  • If regex mode is used, there's an option to add regex flags to control the entire regex that is used in the parse.

  • In regex mode, parse will translate the pattern to a regex. Use RE2 syntax to do the matching, and use numbered captured groups that are handled internally. For example:

      parse kind=regex Col with * <regex1> var1:string <regex2> var2:long
    

    In the parse statement, the regex that will be internally generated by the parse is .*?<regex1>(.*?)<regex2>(\-\d+).

    • * was translated to .*?.

    • string was translated to .*?.

    • long was translated to \-\d+.

Examples

The parse operator provides a streamlined way to extend a table by using multiple extract applications on the same string expression. This result is useful, when the table has a string column that contains several values that you want to break into individual columns. For example, a column that was produced by a developer trace ("printf"/"Console.WriteLine") statement.

In the example below, assume that the column EventText of table Traces contains strings of the form Event: NotifySliceRelease (resourceName={0}, totalSlices= {1}, sliceNumber={2}, lockTime={3}, releaseTime={4}, previousLockTime={5}). The operation will extend the table with six columns: resourceName, totalSlices, sliceNumber, lockTime , releaseTime, previousLockTime, Month, and Day.

let Traces = datatable(EventText:string)
[
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=23, lockTime=02/17/2016 08:40:01, releaseTime=02/17/2016 08:40:01, previousLockTime=02/17/2016 08:39:01)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=15, lockTime=02/17/2016 08:40:00, releaseTime=02/17/2016 08:40:00, previousLockTime=02/17/2016 08:39:00)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=20, lockTime=02/17/2016 08:40:01, releaseTime=02/17/2016 08:40:01, previousLockTime=02/17/2016 08:39:01)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=22, lockTime=02/17/2016 08:41:01, releaseTime=02/17/2016 08:41:00, previousLockTime=02/17/2016 08:40:01)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=16, lockTime=02/17/2016 08:41:00, releaseTime=02/17/2016 08:41:00, previousLockTime=02/17/2016 08:40:00)"
];
Traces  
| parse EventText with * "resourceName=" resourceName ", totalSlices=" totalSlices:long * "sliceNumber=" sliceNumber:long * "lockTime=" lockTime ", releaseTime=" releaseTime:date "," * "previousLockTime=" previousLockTime:date ")" *  
| project resourceName ,totalSlices , sliceNumber , lockTime , releaseTime , previousLockTime
resourceName totalSlices sliceNumber lockTime releaseTime previousLockTime
PipelineScheduler 27 15 02/17/2016 08:40:00 2016-02-17 08:40:00.0000000 2016-02-17 08:39:00.0000000
PipelineScheduler 27 23 02/17/2016 08:40:01 2016-02-17 08:40:01.0000000 2016-02-17 08:39:01.0000000
PipelineScheduler 27 20 02/17/2016 08:40:01 2016-02-17 08:40:01.0000000 2016-02-17 08:39:01.0000000
PipelineScheduler 27 16 02/17/2016 08:41:00 2016-02-17 08:41:00.0000000 2016-02-17 08:40:00.0000000
PipelineScheduler 27 22 02/17/2016 08:41:01 2016-02-17 08:41:00.0000000 2016-02-17 08:40:01.0000000

For regex mode

let Traces = datatable(EventText:string)
[
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=23, lockTime=02/17/2016 08:40:01, releaseTime=02/17/2016 08:40:01, previousLockTime=02/17/2016 08:39:01)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=15, lockTime=02/17/2016 08:40:00, releaseTime=02/17/2016 08:40:00, previousLockTime=02/17/2016 08:39:00)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=20, lockTime=02/17/2016 08:40:01, releaseTime=02/17/2016 08:40:01, previousLockTime=02/17/2016 08:39:01)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=22, lockTime=02/17/2016 08:41:01, releaseTime=02/17/2016 08:41:00, previousLockTime=02/17/2016 08:40:01)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=16, lockTime=02/17/2016 08:41:00, releaseTime=02/17/2016 08:41:00, previousLockTime=02/17/2016 08:40:00)"
];
Traces  
| parse kind = regex EventText with "(.*?)[a-zA-Z]*=" resourceName @", totalSlices=\s*\d+\s*.*?sliceNumber=" sliceNumber:long  ".*?(previous)?lockTime=" lockTime ".*?releaseTime=" releaseTime ".*?previousLockTime=" previousLockTime:date "\\)"  
| project resourceName , sliceNumber , lockTime , releaseTime , previousLockTime
resourceName sliceNumber lockTime releaseTime previousLockTime
PipelineScheduler 15 02/17/2016 08:40:00, 02/17/2016 08:40:00, 2016-02-17 08:39:00.0000000
PipelineScheduler 23 02/17/2016 08:40:01, 02/17/2016 08:40:01, 2016-02-17 08:39:01.0000000
PipelineScheduler 20 02/17/2016 08:40:01, 02/17/2016 08:40:01, 2016-02-17 08:39:01.0000000
PipelineScheduler 16 02/17/2016 08:41:00, 02/17/2016 08:41:00, 2016-02-17 08:40:00.0000000
PipelineScheduler 22 02/17/2016 08:41:01, 02/17/2016 08:41:00, 2016-02-17 08:40:01.0000000

For regex mode using regex flags

If you're interested in getting the resourceName only, use this query:

let Traces = datatable(EventText:string)
[
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=23, lockTime=02/17/2016 08:40:01, releaseTime=02/17/2016 08:40:01, previousLockTime=02/17/2016 08:39:01)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=15, lockTime=02/17/2016 08:40:00, releaseTime=02/17/2016 08:40:00, previousLockTime=02/17/2016 08:39:00)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=20, lockTime=02/17/2016 08:40:01, releaseTime=02/17/2016 08:40:01, previousLockTime=02/17/2016 08:39:01)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=22, lockTime=02/17/2016 08:41:01, releaseTime=02/17/2016 08:41:00, previousLockTime=02/17/2016 08:40:01)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=16, lockTime=02/17/2016 08:41:00, releaseTime=02/17/2016 08:41:00, previousLockTime=02/17/2016 08:40:00)"
];
Traces
| parse kind = regex  EventText with * "resourceName=" resourceName ',' *
| project resourceName
resourceName
PipelineScheduler, totalSlices=27, sliceNumber=23, lockTime=02/17/2016 08:40:01, releaseTime=02/17/2016 08:40:01
PipelineScheduler, totalSlices=27, sliceNumber=15, lockTime=02/17/2016 08:40:00, releaseTime=02/17/2016 08:40:00
PipelineScheduler, totalSlices=27, sliceNumber=20, lockTime=02/17/2016 08:40:01, releaseTime=02/17/2016 08:40:01
PipelineScheduler, totalSlices=27, sliceNumber=22, lockTime=02/17/2016 08:41:01, releaseTime=02/17/2016 08:41:00
PipelineScheduler, totalSlices=27, sliceNumber=16, lockTime=02/17/2016 08:41:00, releaseTime=02/17/2016 08:41:00

You won't get the expected results, since the default mode is greedy. If you have a few records where the resourceName sometimes appears as lower-case and sometimes as upper-case, you may get nulls for some values.

To get the wanted result, run the query with the non-greedy U, and disable case-sensitive i regex flags.

let Traces = datatable(EventText:string)
[
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=23, lockTime=02/17/2016 08:40:01, releaseTime=02/17/2016 08:40:01, previousLockTime=02/17/2016 08:39:01)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=15, lockTime=02/17/2016 08:40:00, releaseTime=02/17/2016 08:40:00, previousLockTime=02/17/2016 08:39:00)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=20, lockTime=02/17/2016 08:40:01, releaseTime=02/17/2016 08:40:01, previousLockTime=02/17/2016 08:39:01)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=22, lockTime=02/17/2016 08:41:01, releaseTime=02/17/2016 08:41:00, previousLockTime=02/17/2016 08:40:01)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=16, lockTime=02/17/2016 08:41:00, releaseTime=02/17/2016 08:41:00, previousLockTime=02/17/2016 08:40:00)"
];
Traces
| parse kind = regex flags = Ui EventText with * "RESOURCENAME=" resourceName ',' *
| project resourceName
resourceName
PipelineScheduler
PipelineScheduler
PipelineScheduler
PipelineScheduler
PipelineScheduler

If the parsed string has newlines, use the flag s, to parse the text.

let Traces = datatable(EventText:string)
[
"Event: NotifySliceRelease (resourceName=PipelineScheduler\ntotalSlices=27\nsliceNumber=23\nlockTime=02/17/2016 08:40:01\nreleaseTime=02/17/2016 08:40:01\npreviousLockTime=02/17/2016 08:39:01)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler\ntotalSlices=27\nsliceNumber=15\nlockTime=02/17/2016 08:40:00\nreleaseTime=02/17/2016 08:40:00\npreviousLockTime=02/17/2016 08:39:00)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler\ntotalSlices=27\nsliceNumber=20\nlockTime=02/17/2016 08:40:01\nreleaseTime=02/17/2016 08:40:01\npreviousLockTime=02/17/2016 08:39:01)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler\ntotalSlices=27\nsliceNumber=22\nlockTime=02/17/2016 08:41:01\nreleaseTime=02/17/2016 08:41:00\npreviousLockTime=02/17/2016 08:40:01)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler\ntotalSlices=27\nsliceNumber=16\nlockTime=02/17/2016 08:41:00\nreleaseTime=02/17/2016 08:41:00\npreviousLockTime=02/17/2016 08:40:00)"
];
Traces
| parse kind=regex flags=s EventText with * "resourceName=" resourceName:string "(.*?)totalSlices=" totalSlices:long "(.*?)lockTime=" lockTime:datetime "(.*?)releaseTime=" releaseTime:datetime "(.*?)previousLockTime=" previousLockTime:datetime "\\)" 
| project-away EventText
resourceName totalSlices lockTime releaseTime previousLockTime
PipelineScheduler
27 2016-02-17 08:40:00.0000000 2016-02-17 08:40:00.0000000 2016-02-17 08:39:00.0000000
PipelineScheduler
27 2016-02-17 08:40:01.0000000 2016-02-17 08:40:01.0000000 2016-02-17 08:39:01.0000000
PipelineScheduler
27 2016-02-17 08:40:01.0000000 2016-02-17 08:40:01.0000000 2016-02-17 08:39:01.0000000
PipelineScheduler
27 2016-02-17 08:41:00.0000000 2016-02-17 08:41:00.0000000 2016-02-17 08:40:00.0000000
PipelineScheduler
27 2016-02-17 08:41:01.0000000 2016-02-17 08:41:00.0000000 2016-02-17 08:40:01.0000000

Relaxed mode

In this example for relaxed mode, totalSlices extended column must be of type long. However, in the parsed string, it has the value nonValidLongValue. In releaseTime extended column, the value nonValidDateTime can't be parsed as datetime. These two extended columns will get the value null while the other ones, such as sliceNumber, still get the correct values.

If you use option kind = simple for the same query below, you'll get null for all extended columns. This option is strict on extended columns, and is the difference between relaxed and simple mode.

Note

In relaxed mode, extended columns can be partially matched.

let Traces = datatable(EventText:string)
[
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=23, lockTime=02/17/2016 08:40:01, releaseTime=nonValidDateTime 08:40:01, previousLockTime=02/17/2016 08:39:01)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=15, lockTime=02/17/2016 08:40:00, releaseTime=nonValidDateTime, previousLockTime=02/17/2016 08:39:00)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=nonValidLongValue, sliceNumber=20, lockTime=02/17/2016 08:40:01, releaseTime=nonValidDateTime 08:40:01, previousLockTime=02/17/2016 08:39:01)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=27, sliceNumber=22, lockTime=02/17/2016 08:41:01, releaseTime=02/17/2016 08:41:00, previousLockTime=02/17/2016 08:40:01)",
"Event: NotifySliceRelease (resourceName=PipelineScheduler, totalSlices=nonValidLongValue, sliceNumber=16, lockTime=02/17/2016 08:41:00, releaseTime=02/17/2016 08:41:00, previousLockTime=02/17/2016 08:40:00)"
];
Traces
| parse kind=relaxed EventText with * "resourceName=" resourceName ", totalSlices=" totalSlices:long * "sliceNumber=" sliceNumber:long * "lockTime=" lockTime ", releaseTime=" releaseTime:date "," * "previousLockTime=" previousLockTime:date ")" *
| project-away EventText
resourceName totalSlices sliceNumber lockTime releaseTime previousLockTime
PipelineScheduler 27 15 02/17/2016 08:40:00 2016-02-17 08:39:00.0000000
PipelineScheduler 27 23 02/17/2016 08:40:01 2016-02-17 08:39:01.0000000
PipelineScheduler 20 02/17/2016 08:40:01 2016-02-17 08:39:01.0000000
PipelineScheduler 16 02/17/2016 08:41:00 2016-02-17 08:41:00.0000000 2016-02-17 08:40:00.0000000
PipelineScheduler 27 22 02/17/2016 08:41:01 2016-02-17 08:41:00.0000000 2016-02-17 08:40:01.0000000