How to use Queue storage from Python

Tip

Try the Microsoft Azure Storage Explorer

Microsoft Azure Storage Explorer is a free, standalone app from Microsoft that enables you to work visually with Azure Storage data on Windows, macOS, and Linux.

Overview

This guide shows you how to perform common scenarios using the Azure Queue storage service. The samples are written in Python and use the Microsoft Azure Storage SDK for Python. The scenarios covered include inserting, peeking, getting, and deleting queue messages, as well as creating and deleting queues. For more information on queues, refer to the [Next Steps] section.

What is Queue Storage?

Azure Queue storage is a service for storing large numbers of messages that can be accessed from anywhere in the world via authenticated calls using HTTP or HTTPS. A single queue message can be up to 64 KB in size, and a queue can contain millions of messages, up to the total capacity limit of a storage account.

Common uses of Queue storage include:

  • Creating a backlog of work to process asynchronously
  • Passing messages from an Azure web role to an Azure worker role

Queue Service Concepts

The Queue service contains the following components:

Queue1

  • URL format: Queues are addressable using the following URL format:
    http://<storage account>.queue.core.windows.net/<queue>

    The following URL addresses a queue in the diagram:

    http://myaccount.queue.core.windows.net/images-to-download

  • Storage Account: All access to Azure Storage is done through a storage account. See Azure Storage Scalability and Performance Targets for details about storage account capacity.

  • Queue: A queue contains a set of messages. All messages must be in a queue. Note that the queue name must be all lowercase. For information on naming queues, see Naming Queues and Metadata.
  • Message: A message, in any format, of up to 64 KB. The maximum time that a message can remain in the queue is 7 days.

Create an Azure storage account

The easiest way to create your first Azure storage account is by using the Azure portal. To learn more, see Create a storage account.

You can also create an Azure storage account by using Azure PowerShell, Azure CLI, or the Storage Resource Provider Client Library for .NET.

If you prefer not to create a storage account at this time, you can also use the Azure storage emulator to run and test your code in a local environment. For more information, see Use the Azure Storage Emulator for Development and Testing.

Download and Install Azure Storage SDK for Python

Azure Storage SDK for Python requires Python 2.7, 3.3, 3.4, 3.5, or 3.6, and comes in 4 different packages: azure-storage-blob, azure-storage-file, azure-storage-table and azure-storage-queue. In this tutorial we are going to use azure-storage-queue package.

Install via PyPi

To install via the Python Package Index (PyPI), type:

pip install azure-storage-queue

Note

If you are upgrading from the Azure Storage SDK for Python version 0.36 or earlier, you will first need to uninstall using pip uninstall azure-storage as we are no longer releasing the Storage SDK for Python in a single package.

For alternative installation methods, visit the Azure Storage SDK for Python on Github.

How To: Create a Queue

The QueueService object lets you work with queues. The following code creates a QueueService object. Add the following near the top of any Python file in which you wish to programmatically access Azure Storage:

from azure.storage.queue import QueueService

The following code creates a QueueService object using the storage account name and account key. Replace 'myaccount' and 'mykey' with your account name and key.

queue_service = QueueService(account_name='myaccount', account_key='mykey')

queue_service.create_queue('taskqueue')

How To: Insert a Message into a Queue

To insert a message into a queue, use the put_message method to create a new message and add it to the queue.

queue_service.put_message('taskqueue', u'Hello World')

How To: Peek at the Next Message

You can peek at the message in the front of a queue without removing it from the queue by calling the peek_messages method. By default, peek_messages peeks at a single message.

messages = queue_service.peek_messages('taskqueue')
for message in messages:
    print(message.content)

How To: Dequeue Messages

Your code removes a message from a queue in two steps. When you call get_messages, you get the next message in a queue by default. A message returned from get_messages becomes invisible to any other code reading messages from this queue. By default, this message stays invisible for 30 seconds. To finish removing the message from the queue, you must also call delete_message. This two-step process of removing a message assures that when your code fails to process a message due to hardware or software failure, another instance of your code can get the same message and try again. Your code calls delete_message right after the message has been processed.

messages = queue_service.get_messages('taskqueue')
for message in messages:
    print(message.content)
    queue_service.delete_message('taskqueue', message.id, message.pop_receipt)

There are two ways you can customize message retrieval from a queue. First, you can get a batch of messages (up to 32). Second, you can set a longer or shorter invisibility timeout, allowing your code more or less time to fully process each message. The following code example uses the get_messages method to get 16 messages in one call. Then it processes each message using a for loop. It also sets the invisibility timeout to five minutes for each message.

messages = queue_service.get_messages('taskqueue', num_messages=16, visibility_timeout=5*60)
for message in messages:
    print(message.content)
    queue_service.delete_message('taskqueue', message.id, message.pop_receipt)        

How To: Change the Contents of a Queued Message

You can change the contents of a message in-place in the queue. If the message represents a work task, you could use this feature to update the status of the work task. The code below uses the update_message method to update a message. The visibility timeout is set to 0, meaning the message appears immediately and the content is updated.

messages = queue_service.get_messages('taskqueue')
for message in messages:
    queue_service.update_message('taskqueue', message.id, message.pop_receipt, 0, u'Hello World Again')

How To: Get the Queue Length

You can get an estimate of the number of messages in a queue. The get_queue_metadata method asks the queue service to return metadata about the queue, and the approximate_message_count. The result is only approximate because messages can be added or removed after the queue service responds to your request.

metadata = queue_service.get_queue_metadata('taskqueue')
count = metadata.approximate_message_count

How To: Delete a Queue

To delete a queue and all the messages contained in it, call the delete_queue method.

queue_service.delete_queue('taskqueue')

Next Steps

Now that you've learned the basics of Queue storage, follow these links to learn more.