Edit

Share via

Send data to Azure Blob Storage from a Data Processor pipeline

  • Article

Important

Azure IoT Operations Preview – enabled by Azure Arc is currently in PREVIEW. You shouldn't use this preview software in production environments.

See the Supplemental Terms of Use for Microsoft Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

Use the Azure Blob Storage destination to write unstructured data to Azure Blob Storage for storage and analysis.

Prerequisites

To configure and use this Azure Blob Storage destination pipeline stage, you need:

  • A deployed instance of Data Processor.
  • An Azure Blob Storage account.

Configure the destination stage

The Azure Blob Storage destination stage JSON configuration defines the details of the stage. To author the stage, you can either interact with the form-based UI, or provide the JSON configuration on the Advanced tab:

Field Type Description Required? Default Example
accountName string The name the Azure Blob Storage account. Yes myBlobStorageAccount
containerName string The name of container created in the storage account to store the blobs. Yes mycontainer
authentication string Authentication information to connect to the storage account. One of servicePrincipal, systemAssignedManagedIdentity, and accessKey. Yes See the sample configuration.
format Object. Formatting information for data. All types are supported. Yes {"type": "json"}
blobPath Templates Template string that identifies the path to write files to. All the template components shown in the default are required. No {{{instanceId}}}/{{{pipelineId}}}/{{{partitionId}}}/{{{YYYY}}}/{{{MM}}}/{{{DD}}}/{{{HH}}}/{{{mm}}}/{{{fileNumber}}} {{{instanceId}}}/{{{pipelineId}}}/{{{partitionId}}}/{{{YYYY}}}/{{{MM}}}/{{{DD}}}/{{{HH}}}/{{{mm}}}/{{{fileNumber}}}.xyz
batch Batch How to batch data before writing it to Blob Storage. No {"time": "60s"} {"time": "60s"}
retry Retry The retry mechanism to use when a Blob Storage operation fails. No (empty) {"type": "fixed"}

Sample configuration

The following JSON shows a sample configuration for the Azure Blob Storage destination stage:

{
    "displayName": "Sample blobstorage output",
    "description": "An example blobstorage output stage",
    "type": "output/blobstorage@v1",
    "accountName": "myStorageAccount",
    "containerName": "mycontainer",
    "blobPath": "{{{instanceId}}}/{{{pipelineId}}}/{{{partitionId}}}/{{{YYYY}}}/{{{MM}}}/{{{DD}}}/{{{HH}}}/{{{mm}}}/{{{fileNumber}}}",
    "authentication": {
        "type": "systemAssignedManagedIdentity"
    },
    "format": {
        "type": "json"
    },
    "batch": {
        "time": "60s",
        "path": ".payload"
    },
    "retry": {
        "type": "fixed"
    }
}

Related content