Reacting to Blob storage events

Azure Storage events allow applications to react to the creation and deletion of blobs using modern serverless architectures. It does so without the need for complicated code or expensive and inefficient polling services. Instead, events are pushed through Azure Event Grid to subscribers such as Azure Functions, Azure Logic Apps, or even to your own custom http listener, and you only pay for what you use.

Blob storage events are reliably sent to the Event grid service which provides reliable delivery services to your applications through rich retry policies and dead-letter delivery.

Common Blob storage event scenarios include image or video processing, search indexing, or any file-oriented workflow. Asynchronous file uploads are a great fit for events. When changes are infrequent, but your scenario requires immediate responsiveness, event-based architecture can be especially efficient.

Take a look at Route Blob storage events to a custom web endpoint - CLI or Route Blob storage events to a custom web endpoint - PowerShell for a quick example.

Event Grid Model

Blob storage accounts

Blob storage events are available in general-purpose v2 storage accounts and Blob storage accounts. General-purpose v2 storage accounts support all features for all storage services, including Blobs, Files, Queues, and Tables. A Blob storage account is a specialized storage account for storing your unstructured data as blobs (objects) in Azure Storage. Blob storage accounts are like general-purpose storage accounts and share all the great durability, availability, scalability, and performance features that you use today including 100% API consistency for block blobs and append blobs. For more information, see Azure storage account overview.

Available Blob storage events

Event grid uses event subscriptions to route event messages to subscribers. Blob storage event subscriptions can include two types of events:

Event Name Description
Microsoft.Storage.BlobCreated Fired when a blob is created or replaced through the PutBlob, PutBlockList, or CopyBlob operations
Microsoft.Storage.BlobDeleted Fired when a blob is deleted through a DeleteBlob operation

Event Schema

Blob storage events contain all the information you need to respond to changes in your data. You can identify a Blob storage event because the eventType property starts with "Microsoft.Storage". Additional information about the usage of Event Grid event properties is documented in Event Grid event schema.

Property Type Description
topic string Full Azure Resource Manager id of the storage account that emits the event.
subject string The relative resource path to the object that is the subject of the event, using the same extended Azure Resource Manager format that we use to describe storage accounts, services, and containers for Azure RBAC. This format includes a case-preserving blob name.
eventTime string Date/time that the event was generated, in ISO 8601 format
eventType string "Microsoft.Storage.BlobCreated" or "Microsoft.Storage.BlobDeleted"
Id string Unique identifier if this event
dataVersion string The schema version of the data object.
metadataVersion string The schema version of top-level properties.
data object Collection of blob storage-specific event data
data.contentType string The content type of the blob, as would be returned in the Content-Type header from the blob
data.contentLength number The size of the blob as in integer representing a number of bytes, as would be returned in the Content-Length header from the blob. Sent with BlobCreated event, but not with BlobDeleted.
data.url string The url of the object that is the subject of the event
data.eTag string The etag of the object when this event fired. Not available for the BlobDeleted event.
data.api string The name of the api operation that triggered this event. For BlobCreated events, this value is "PutBlob", "PutBlockList", or "CopyBlob". For BlobDeleted events, this value is "DeleteBlob". These values are the same api names that are present in the Azure Storage diagnostic logs. See Logged Operations and Status Messages.
data.sequencer string An opaque string value representing the logical sequence of events for any particular blob name. Users can use standard string comparison to understand the relative sequence of two events on the same blob name.
data.requestId string Service-generated request id for the storage API operation. Can be used to correlate to Azure Storage diagnostic logs using the "request-id-header" field in the logs and is returned from initiating API call in the 'x-ms-request-id' header. See Log Format.
data.clientRequestId string Client-provided request id for the storage API operation. Can be used to correlate to Azure Storage diagnostic logs using the "client-request-id" field in the logs, and can be provided in client requests using the "x-ms-client-request-id" header. See Log Format.
data.storageDiagnostics object Diagnostic data occasionally included by the Azure Storage service. When present, should be ignored by event consumers.
data.blobType string The type of the blob. Valid values are either "BlockBlob" or "PageBlob".

Here is an example of a BlobCreated event:

[{
  "topic": "/subscriptions/319a9601-1ec0-0000-aebc-8fe82724c81e/resourceGroups/testrg/providers/Microsoft.Storage/storageAccounts/myaccount",
  "subject": "/blobServices/default/containers/testcontainer/blobs/file1.txt",
  "eventType": "Microsoft.Storage.BlobCreated",
  "eventTime": "2017-08-16T01:57:26.005121Z",
  "id": "602a88ef-0001-00e6-1233-1646070610ea",
  "data": {
    "api": "PutBlockList",
    "clientRequestId": "799304a4-bbc5-45b6-9849-ec2c66be800a",
    "requestId": "602a88ef-0001-00e6-1233-164607000000",
    "eTag": "0x8D4E44A24ABE7F1",
    "contentType": "text/plain",
    "contentLength": 447,
    "blobType": "BlockBlob",
    "url": "https://myaccount.blob.core.windows.net/testcontainer/file1.txt",
    "sequencer": "00000000000000EB000000000000C65A",
  },
  "dataVersion": "",
  "metadataVersion": "1"
}]

For more information, see Blob storage events schema.

Filtering events

Blob event subscriptions can be filtered based on the event type and by the container name and blob name of the object that was created or deleted. Filters can be applied to event subscriptions either during the creation of the event subscription or at a later time. Subject filters in Event Grid work based on "begins with" and "ends with" matches, so that events with a matching subject are delivered to the subscriber.

The subject of Blob storage events uses the format:

/blobServices/default/containers/<containername>/blobs/<blobname>

To match all events for a storage account, you can leave the subject filters empty.

To match events from blobs created in a set of containers sharing a prefix, use a subjectBeginsWith filter like:

/blobServices/default/containers/containerprefix

To match events from blobs created in specific container, use a subjectBeginsWith filter like:

/blobServices/default/containers/containername/

To match events from blobs created in specific container sharing a blob name prefix, use a subjectBeginsWith filter like:

/blobServices/default/containers/containername/blobs/blobprefix

To match events from blobs created in specific container sharing a blob suffix, use a subjectEndsWith filter like ".log" or ".jpg". For more information, see Event Grid Concepts.

Practices for consuming events

Applications that handle Blob storage events should follow a few recommended practices:

  • As multiple subscriptions can be configured to route events to the same event handler, it is important not to assume events are from a particular source, but to check the topic of the message to ensure that it comes from the storage account you are expecting.
  • Similarly, check that the eventType is one you are prepared to process, and do not assume that all events you receive will be the types you expect.
  • As messages can arrive out of order and after some delay, use the etag fields to understand if your information about objects is still up-to-date. Also, use the sequencer fields to understand the order of events on any particular object.
  • Use the blobType field to understand what type of operations are allowed on the blob, and which client library types you should use to access the blob. Valid values are either BlockBlob or PageBlob.
  • Use the url field with the CloudBlockBlob and CloudAppendBlob constructors to access the blob.
  • Ignore fields you don't understand. This practice will help keep you resilient to new features that might be added in the future.

Next steps

Learn more about Event Grid and give Blob storage events a try: