Edit

Share via

Filter data in an Azure IoT Data Processor Preview pipeline

  • Article

Important

Azure IoT Operations Preview – enabled by Azure Arc is currently in PREVIEW. You shouldn't use this preview software in production environments.

See the Supplemental Terms of Use for Microsoft Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

Use a filter stage to filter out messages that you don't need for further processing in the pipeline. The stage emits the original message to the filter stage unchanged if the filter criteria is met, otherwise the stage drops the message from the pipeline.

  • Each pipeline partition filters out messages independently of other partitions.
  • The output of the filter stage is the original message if the stage doesn't filter it out.

Prerequisites

To configure and use a filter pipeline stage, you need a deployed instance of Azure IoT Data Processor Preview.

Configure the stage

The filter stage JSON configuration defines the details of the stage. To author the stage, you can either interact with the form-based UI, or provide the JSON configuration on the Advanced tab:

Name Value Required Default Example
Display name A name to show in the Data Processor UI. Yes - Filter1
Description A user-friendly description of what the filter stage does. No - Filter out anomalies
Query The jq expression Yes - .payload.temperature > 0 and .payload.pressure < 50

jq expression

Filter queries in Data Processor use the jq language to define the filter condition:

  • The jq provided in the query must be syntactically valid.
  • The result of the filter query must be a boolean value.
  • Messages that evaluate to true are emitted unchanged from the filter stage to subsequent stages for further processing. Messages that evaluate to false are dropped from the pipeline.
  • All messages for which the filter doesn't return a boolean result are treated as an error case and dropped from the pipeline.
  • The filter stage adheres to the same restriction on jq usage as defined in the jq expression guide.

When you create a filter query to use in the filter stage:

  • Test your filter query with your messages to make sure a boolean result is returned.
  • Configure the filter query based on how the message arrives at the filter stage.
  • To learn more about building your filter expressions, see thejq expressions guide.

Sample configuration

The following JSON example shows a complete filter stage configuration:

{ 
    "displayName": "Filter name", 
    "description": "Filter description", 
    "query": "(.properties.responseTopic | contains(\"bar\")) or (.properties.responseTopic | contains(\"baz\")) and (.payload | has(\"temperature\")) and (.payload.temperature > 0)"
}

This filter checks for messages where .properties.responseTopic contains bar or baz and the message payload has a property called temperature with a value greater than 0.

Related content