How to use Service Bus queues with Python
This article describes how to use Service Bus queues. The samples are written in Python and use the Python Azure Service Bus package. The scenarios covered include creating queues, sending and receiving messages, and deleting queues.
What are Service Bus queues?
Service Bus queues support a brokered messaging communication model. When using queues, components of a distributed application do not communicate directly with each other; instead they exchange messages via a queue, which acts as an intermediary (broker). A message producer (sender) hands off a message to the queue and then continues its processing. Asynchronously, a message consumer (receiver) pulls the message from the queue and processes it. The producer does not have to wait for a reply from the consumer in order to continue to process and send further messages. Queues offer First In, First Out (FIFO) message delivery to one or more competing consumers. That is, messages are typically received and processed by the receivers in the order in which they were added to the queue, and each message is received and processed by only one message consumer.
Service Bus queues are a general-purpose technology that can be used for a wide variety of scenarios:
- Communication between web and worker roles in a multi-tier Azure application.
- Communication between on-premises apps and Azure-hosted apps in a hybrid solution.
- Communication between components of a distributed application running on-premises in different organizations or departments of an organization.
Using queues enables you to scale your applications more easily, and enable more resiliency to your architecture.
Create a namespace in the Azure portal
To begin using Service Bus messaging entities in Azure, you must first create a namespace with a name that is unique across Azure. A namespace provides a scoping container for addressing Service Bus resources within your application.
To create a namespace:
- Sign in to the Azure portal
In the left navigation pane of the portal, select + Create a resource, select Integration, and then select Service Bus.
In the Create namespace dialog, do the following steps:
- Enter a name for the namespace. The system immediately checks to see if the name is available.
- Select the pricing tier (Basic, Standard, or Premium) for the namespace. If you want to use topics and subscriptions, choose either Standard or Premium. Topics/subscriptions are not supported in the Basic pricing tier.
- If you selected the Premium pricing tier, follow these steps:
- Specify the number of messaging units. The premium tier provides resource isolation at the CPU and memory level so that each workload runs in isolation. This resource container is called a messaging unit. A premium namespace has least one messaging unit. You can select 1, 2, or 4 messaging units for each Service Bus Premium namespace. For more information, see Service Bus Premium Messaging.
- Specify whether you want to make the namespace zone redundant. The zone redundancy provides enhanced availability by spreading replicas across availability zones within one region at no additional cost. For more information, see Availability zones in Azure.
- For Subscription, choose an Azure subscription in which to create the namespace.
- For Resource group, choose an existing resource group in which the namespace will live, or create a new one.
- For Location, choose the region in which your namespace should be hosted.
Select Create. The system now creates your namespace and enables it. You might have to wait several minutes as the system provisions resources for your account.
Confirm that the service bus namespace is deployed successfully. To see the notifications, select the bell icon (Alerts) on the toolbar. Select the name of the resource group in the notification as shown in the image. You see the resource group that contains the service bus namespace.
On the Resource group page for your resource group, select your service bus namespace.
You see the home page for your service bus namespace.
Get the connection string
Creating a new namespace automatically generates an initial Shared Access Signature (SAS) rule with an associated pair of primary and secondary keys that each grant full control over all aspects of the namespace. See Service Bus authentication and authorization for information about how to create rules with more constrained rights for regular senders and receivers. To copy the primary and secondary keys for your namespace, follow these steps:
- Click All resources, then click the newly created namespace name.
- In the namespace window, click Shared access policies.
In the Shared access policies screen, click RootManageSharedAccessKey.
In the Policy: RootManageSharedAccessKey window, click the copy button next to Primary Connection String, to copy the connection string to your clipboard for later use. Paste this value into Notepad or some other temporary location.
- Repeat the previous step, copying and pasting the value of Primary key to a temporary location for later use.
Create a queue
The ServiceBusService object enables you to work with queues. Add the following code near the top of any Python file in which you wish to programmatically access Service Bus:
from azure.servicebus import ServiceBusService, Message, Queue
The following code creates a ServiceBusService object. Replace
sharedaccesskey with your namespace, shared access signature (SAS) key name, and value.
bus_service = ServiceBusService( service_namespace='mynamespace', shared_access_key_name='sharedaccesskeyname', shared_access_key_value='sharedaccesskey')
The values for the SAS key name and value can be found in the Azure portal connection information, or in the Visual Studio Properties pane when selecting the Service Bus namespace in Server Explorer (as shown in the previous section).
create_queue method also supports additional options, which enable you to override default queue settings such as message time to live (TTL) or maximum queue size. The following example sets the maximum queue size to 5 GB, and the TTL value to 1 minute:
queue_options = Queue() queue_options.max_size_in_megabytes = '5120' queue_options.default_message_time_to_live = 'PT1M' bus_service.create_queue('taskqueue', queue_options)
Send messages to a queue
To send a message to a Service Bus queue, your application calls the
send_queue_message method on the ServiceBusService object.
The following example demonstrates how to send a test message to the queue named
msg = Message(b'Test Message') bus_service.send_queue_message('taskqueue', msg)
Service Bus queues support a maximum message size of 256 KB in the Standard tier and 1 MB in the Premium tier. The header, which includes the standard and custom application properties, can have a maximum size of 64 KB. There is no limit on the number of messages held in a queue but there is a cap on the total size of the messages held by a queue. This queue size is defined at creation time, with an upper limit of 5 GB. For more information about quotas, see Service Bus quotas.
Receive messages from a queue
Messages are received from a queue using the
receive_queue_message method on the ServiceBusService object:
msg = bus_service.receive_queue_message('taskqueue', peek_lock=False) print(msg.body)
Messages are deleted from the queue as they are read when the parameter
peek_lock is set to False. You can read (peek) and lock the message without deleting it from the queue by setting the parameter
peek_lock to True.
The behavior of reading and deleting the message as part of the receive operation is the simplest model, and works best for scenarios in which an application can tolerate not processing a message in the event of a failure. To understand this, consider a scenario in which the consumer issues the receive request and then crashes before processing it. Because Service Bus will have marked the message as being consumed, then when the application restarts and begins consuming messages again, it will have missed the message that was consumed prior to the crash.
peek_lock parameter is set to True, the receive becomes a two stage operation, which makes it possible to support applications that cannot tolerate missing messages. When Service Bus receives a request, it finds the next message to be consumed, locks it to prevent other consumers receiving it, and then returns it to the application. After the application finishes processing the message (or stores it reliably for future processing), it completes the second stage of the receive process by calling the delete method on the Message object. The delete method will mark the message as being consumed and remove it from the queue.
msg = bus_service.receive_queue_message('taskqueue', peek_lock=True) print(msg.body) msg.delete()
How to handle application crashes and unreadable messages
Service Bus provides functionality to help you gracefully recover from errors in your application or difficulties processing a message. If a receiver application is unable to process the message for some reason, then it can call the unlock method on the Message object. This will cause Service Bus to unlock the message within the queue and make it available to be received again, either by the same consuming application or by another consuming application.
There is also a timeout associated with a message locked within the queue, and if the application fails to process the message before the lock timeout expires (e.g., if the application crashes), then Service Bus will unlock the message automatically and make it available to be received again.
In the event that the application crashes after processing the message but before the delete method is called, then the message will be redelivered to the application when it restarts. This is often called At Least Once Processing, that is, each message will be processed at least once but in certain situations the same message may be redelivered. If the scenario cannot tolerate duplicate processing, then application developers should add additional logic to their application to handle duplicate message delivery. This is often achieved using the MessageId property of the message, which will remain constant across delivery attempts.
Now that you have learned the basics of Service Bus queues, see these articles to learn more.
We'd love to hear your thoughts. Choose the type you'd like to provide:
Our feedback system is built on GitHub Issues. Read more on our blog.